Is AI Resume Screening Biased? An Honest Answer

Yes — AI resume screening can discriminate, and there is published evidence that it does. No vendor, including us, can honestly sell you a bias-free system. What a serious vendor can do is name where bias enters, hand you the controls that reduce it, and let you measure the damage yourself. This post does that, and tells you what Indian law does and does not currently require.

The evidence, before the excuses

In April 2025, Brookings published research by Kyra Wilson and Aylin Caliskan that simulated resume screening with three open large language models. They took 554 real resumes, attached 80 names signalling different race and gender identities, and ranked them against 571 job descriptions across nine occupations. White-associated names were preferred 85.1% of the time against Black-associated names, which were preferred 8.6% of the time. Male-associated names were preferred 51.9% of the time against female-associated names at 11.1%. In the intersectional test, Black male names were selected 0% of the time against white male names.

That is not a hypothetical. That is the default behaviour of an off-the-shelf embedding model asked to rank candidates against a JD — which is exactly what most "AI screening" is under the hood.

It is also not new. Reuters reported in October 2018 that Amazon scrapped an internal recruiting model after finding it penalised resumes containing the word "women's" — as in "women's chess club captain." The model had learned from ten years of Amazon's own hiring, and that history was male. The model was working correctly. It was the objective that was wrong.

Note what both cases have in common: nobody typed in a rule that said "prefer men." Bias in screening is almost never a rule. It is a correlation the model found and you did not.

Where bias actually enters

1. The criteria you wrote

Bias enters before any model runs. Job ads carry gendered wording — Gaucher, Friesen and Kay documented this in the Journal of Personality and Social Psychology in 2011: masculine-coded language in job adverts ("dominant," "competitive," "aggressive") reduces how much women want the job, without changing their ability to do it. Feed that JD into a semantic matcher and it will rank resumes that mirror the same register more highly.

Then there are the criteria you added yourself because they "filter for quality":

• College tier. "Tier-1 institute only" is not a skill. In India it tracks coaching access, family income, urban location, and school medium — and, through all of those, caste and community.
• Career gaps. A hard "no unexplained gaps" filter is a maternity filter with extra steps.
• English polish. Screening on fluency of the resume's prose, for a role that does not require writing, is screening on which language you were schooled in.
• Pincode or hometown. Communities cluster geographically in India. A location filter can be a caste filter you did not intend.

The single most effective control here is boring: derive your must-have criteria from the actual job description, not from your instinct about "the kind of person who does well here." Our keyword scanner exists to pull the concrete skills out of a JD; the point of doing that is that a skill is auditable and "IIT preferred" is not.

2. The model's training data

If a screening tool learns from your historical hires, it will reproduce your historical hires. That is what happened at Amazon. Any vendor that pitches "it learns from your best performers" is pitching, in plain language, "it will find more people like the ones you already have." If your last three years of engineering hires are 90% male, that is now the target profile.

Ask the vendor directly: does the ranking model train on our past hiring outcomes? Yes or no? "No" is a legitimate architecture. "Yes, that's the magic" is a red flag wearing a lanyard.

3. The resume itself

This is the part US-written articles miss entirely, and it matters most in India. The standard Indian CV — the biodata format still taught in most colleges — routinely carries a photograph, date of birth, gender, marital status, father's or husband's name, native place, and sometimes religion, category or mother tongue. None of that is on a typical American resume. All of it is a direct read of protected attributes.

And even if you strip every field, the surname remains. In India, surname is a strong signal of caste, region and religion. Stripping the first name — the "blind screening" that Western tooling offers — does very little here. The Brookings/UW result showed models responding to names alone; Indian surnames carry at least as much identity information as the American names in that study.

Practical implication: redaction has to be aggressive, and even aggressive redaction is incomplete. Strip photo, name, DOB, gender, marital status, address. Accept that college, employer names and the writing style still leak. Redaction reduces the signal. It does not remove it.

4. The feedback loop

Screening output becomes hiring input becomes training data. Run a slightly skewed model for four quarters, retrain on the resulting hires, and the skew is now the ground truth. This is the failure mode that turns a small bias into an entrenched one, and it is invisible from inside a single hiring cycle. The only defence is measuring outcomes over time — which brings us to the one thing you can do without a vendor, a lawyer, or a data scientist.

Measure your own funnel. It takes a spreadsheet.

The US EEOC's Uniform Guidelines have used the four-fifths rule for decades, and NYC's Local Law 144 requires exactly this calculation from any employer using an automated employment decision tool: compute the selection rate for each group, divide by the selection rate of the most-selected group, and flag anything below 0.8.

Worked example — the numbers below are invented purely to show the arithmetic, not data from any real funnel:

0.55 is well under 0.8. That funnel has an adverse impact problem regardless of what the vendor's marketing page says, and you found it with two columns and a division.

Do this monthly, per role. Track sex at minimum. If you collect it, track the categories that matter for your organisation. If the ratio drops below 0.8, the model is not the only suspect — your JD and your criteria are suspects too — but you now have a number instead of an argument.

What the law actually says

The EU: employment screening is explicitly high-risk

The EU Artificial Intelligence Act (Regulation (EU) 2024/1689) does not hedge. Under Article 6(2) and Annex III, point 4(a), high-risk AI systems include those "intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates."

That is a resume screener, described precisely.

One important correction to almost every article you will read on this: the Digital Omnibus package agreed in 2026 pushed the application date for stand-alone Annex III high-risk obligations from 2 August 2026 to 2 December 2027 (per Gibson Dunn's analysis of the omnibus agreement). The classification did not change — recruitment is still high-risk. Only the deadline moved. If a vendor tells you they are "AI Act compliant" as a finished fact today, ask what they think the current deadline is.

This is not academic for Indian firms. The AI Act reaches providers and deployers outside the EU where the system's output is used in the Union. If you screen candidates for EU roles, or your product screens for European clients, you are in scope.

The US: audit obligations already exist

NYC Local Law 144 (of 2021) prohibits using an automated employment decision tool unless it has had an independent bias audit within the past year, the audit summary is published, and candidates are notified. The audit must calculate selection rates and impact ratios by sex, by race/ethnicity, and intersectionally. NYC's DCWP can levy civil penalties per day of violation. Illinois and Colorado have moved in similar directions.

India: almost nothing, and you should plan accordingly

Here is the honest position. India has no statute that holds a private employer accountable for algorithmic bias in hiring. The International Bar Association's own review of AI in Indian workplaces states there is "no specific legal architecture that holds employers accountable for algorithmic bias or enforces transparency, fairness audits or human oversight in automated hiring systems."

Concretely:

• The Constitution's equality provisions (Articles 14, 15, 16) bind the State and public employment. They are not a general anti-discrimination code for private hiring.
• Subject-specific statutes exist — the Rights of Persons with Disabilities Act 2016, the Transgender Persons (Protection of Rights) Act 2019, the Maternity Benefit Act, equal-remuneration provisions — but they are narrow, and none was written with a ranking model in mind.
• The DPDP Act, 2023 governs personal data. It does not give candidates a GDPR Article 22-style right to contest a purely automated decision.
• MeitY's India AI Governance Guidelines (November 2025) are explicitly voluntary. No fine attaches to ignoring them.

So: nobody will fine you. That is precisely why this is a governance decision and not a compliance one. The two forces that will actually reach you are (a) your EU and US clients, who are already in scope and will push the obligation down their supply chain, and (b) a candidate with a screenshot and a Twitter account.

The controls to demand from any vendor

Take this list into the demo. The useful part is not the question — it is knowing what a real answer sounds like.

1. "Does your ranking model train on our past hiring decisions?" Good: "No. Scores are computed against criteria derived from the job description you supply." Bad: "Yes — it learns your ideal candidate profile from your top performers."

2. "What fields do you redact before scoring, and what remains visible to the model?" Good: A specific list, plus a candid admission that surname, college and writing style still leak signal. Bad: "Our AI is blind to demographics." No system that reads a resume is blind to demographics.

3. "Have you published an independent bias audit? Can I see the impact ratios?" Good: A published audit, or an honest "not yet, and here is our timeline." Bad: "Our algorithm is unbiased by design." There is no such design.

4. "Can I see, per candidate, the reason for the score — and can I override it?" If a recruiter cannot reconstruct why candidate #47 was ranked below #12, you cannot defend the decision, to a client or to a court.

5. "Do you retain decision logs, and for how long?" An adverse-impact question asked six months later is unanswerable without logs.

6. "Is our candidate data used to train your models?" This should be a contractual "no," not a settings toggle.

7. Human review at the rejection boundary, not just the top of the list. Everyone reviews the top ten. Bias lives at the cut line — candidates 25 to 40. Sample and review that band.

Where we stand

ShortlistAI is a resume screening tool. We are not going to tell you it is bias-free, because that claim is not available to any vendor and anyone making it is either uninformed or lying to you.

What we will say plainly: we have not published an independent third-party bias audit. When we do, it will be linked here. If you are evaluating us, hold us to the seven questions above, and where we cannot meet a line item today, we would rather tell you than pretend. A screening vendor that will not describe its own failure modes has not thought about them.

FAQ

Is it legal to use AI resume screening in India? Yes. There is currently no Indian statute prohibiting it, requiring a bias audit, or mandating that you disclose its use to candidates. That is a gap in the law, not a certificate of safety — and it does not protect you if you screen for EU or NYC roles, where obligations already attach.

Does removing names from resumes fix the bias? No. It reduces one signal. In India, surname alone carries caste, region and religion; college, employer, pincode, career gaps and writing style all continue to leak identity after the name is gone. Blind screening is a control, not a cure.

We're an Indian company. Does the EU AI Act really apply to us? It can. The AI Act reaches providers and deployers established outside the EU where the output of the system is used within the Union. If you screen candidates for EU-based roles, or you supply screening to European clients, assume you are in scope and read Annex III, point 4(a). The high-risk obligations now apply from 2 December 2027 after the Digital Omnibus deferral — the classification itself is unchanged.

Isn't a human recruiter biased too? Yes, demonstrably. But the comparison cuts both ways. A biased human reviews 60 resumes; a biased model reviews 6,000 and applies the same distortion to every one, consistently, at speed. The genuine advantage of the machine is not that it is fairer — it is that it is measurable. You can compute an impact ratio on a model's output. You cannot compute one on a hiring manager's gut. Use that advantage: measure the funnel, monthly, and act on the number.