Shortlist quotas require organizations to include a minimum number (or fraction) of candidates with a given demographic identity (e.g., women) in their pool of finalists for a job and are widely used to increase diversity. But do they work? They may boost the diversity of hires by altering the composition of choice sets, or by signaling that diversity is valued. However, shortlist quotas may not affect final hiring choices since they are non-binding; they may also spur reactance. In four, pre-registered, incentive-compatible experiments including a shortlist and a hiring stage in a real hiring game, we test the impact of a shortlist quota requiring the inclusion of at least one woman in the finalist set considered for an opening. We also test whether informing hiring managers that finalist sets were required to include at least one woman undermines or enhances quotas’ effectiveness. We find that shortlist quotas both (mechanically) increase the number of women included in finalist sets and significantly increase the number of finalist sets with more than one woman (by ~5.2%). Moreover, ~11-15% more women are eventually hired in games involving a shortlist quota, and this effect is larger (~15-19% more women are hired) when final hiring managers know a quota was implemented at the shortlist stage. Shortlist quotas boost perceptions that others choose women and that an organization believes gender diversity is valuable, both of which mediate the decision to select women at the shortlist stage and when making a final hire.