Who Gets Flagged? When AI Detectors Learn Our Biases

Classroom.

A student submits an essay. A detector returns a score. Someone in authority reads that score as evidence. The student now has to prove that their own words are, in fact, their own.

This is the point where AI-text detection stops being a technical widget and becomes an institutional decision system. The question is no longer just “Can this model distinguish AI-generated text from human writing?” It is “Which humans does it fail to recognize as human?”

That is the uncomfortable center of BAID, a benchmark proposed by Priyam Basu, Yunfeng Zhang, and Vipul Raheja for assessing bias in AI-generated text detectors.¹ The paper’s contribution is not another detector leaderboard. Leaderboards are easy. Everyone loves a tidy F1 score. It looks scientific, and it fits nicely in procurement slides, which is how many bad decisions achieve adulthood.

BAID does something more useful: it disaggregates detector behavior across seven bias dimensions and 41 subgroups, using 208,166 paired human-written and AI-generated documents. It then tests four detectors—Desklib, E5-small, Radar, and ZipPy—to show how performance that appears acceptable in aggregate can collapse for particular writing styles, dialects, age groups, topics, and demographic categories.

The central misconception is simple: if an AI detector has decent overall accuracy, it is safe enough to deploy.

BAID says: no. Not if the detector’s errors concentrate on particular groups. Not if informal writing is treated as suspicious. Not if dialectal English becomes a false-positive trap. Not if a student, applicant, worker, journalist, or platform user has to carry the burden of proving that their linguistic style is not machine output.

BAID turns detector evaluation into subgroup accounting

Most AI-text detector evaluations ask whether a detector can separate human-authored text from machine-authored text under reasonably standard conditions. BAID changes the accounting unit.

Instead of treating “human writing” as one stable category, the benchmark asks whether detectors behave differently across seven dimensions:

The dataset design matters. BAID contains human-written documents and AI-generated counterparts that preserve the original meaning while reflecting subgroup-specific writing styles. The AI versions are generated with GPT-4.1 and Claude Sonnet 3.7, then filtered for quality, including semantic-alignment checks using an embedding similarity threshold of 0.85.

That pairing is useful, but the authors make an important methodological choice: fairness evaluation is centered on human-written texts. This is the right move. A synthetic “AAVE-style” or “GenZ-style” rewrite is not the same as real human authorship from a real social group. It can test detector calibration under controlled stylistic variation. It cannot prove fairness toward actual people.

That distinction becomes important later. The paper’s main evidence comes from subgroup-level performance on human-written texts. The appendix results on AI-generated samples serve a different purpose: they help explain calibration and sensitivity to synthetic style, not demographic fairness itself.

The detector list is small, but architecturally useful

BAID evaluates four detectors:

This is not a universal detector census. The paper does not cover every commercial detector, every multilingual system, or every hybrid architecture. But the four systems are different enough to expose the bigger point: bias is not only a dataset problem. It is also an architecture problem.

Neural detectors, compression detectors, and adversarially trained detectors fail differently. That difference is exactly what a procurement team should care about. A detector that is “best overall” may still be worst for the population most exposed to its decisions.

Demographics: standardized writing does not eliminate uneven recall

The demographic results are the least sensational and the most institutionally relevant.

Across many demographic subgroups, Desklib performs strongly on human-written text. For gender, its recall is 0.84 for female writers and 0.85 for male writers, with F1 around 0.91–0.92. For many race and ethnicity groups, it also stays relatively high. At first glance, this looks reassuring.

Then the subgroup table begins doing what subgroup tables are supposed to do: ruining the comfort of averages.

For American Indian/Alaskan Native writers, Desklib recall is 0.65, compared with 0.82 for Asian/Pacific Islander writers, 0.87 for Black/African American writers, 0.85 for Hispanic/Latino writers, and 0.86 for White writers. The American Indian/Alaskan Native subgroup is much smaller, with 184 samples, so this should be read carefully rather than theatrically. But the signal is still useful: a detector can look stable across broad demographics while showing a sharp pocket of weakness in a smaller subgroup.

The English Language Learner result is also important because it connects BAID to earlier concerns about AI detectors penalizing non-native English writing. Desklib recall is 0.77 for ELL writers and 0.85 for non-ELL writers. E5-small performs weakly in both cases, with recall of 0.20 for ELL writers and 0.24 for non-ELL writers, while ZipPy stays low at 0.17 and 0.23.

The lesson is not “every detector is equally biased against ELL writers.” The sharper lesson is that ELL status is not the only fairness risk. BAID broadens the concern from one known failure case into a wider audit surface.

Disability status shows another kind of unevenness. E5-small recall is 0.19 for writers with disability status marked “yes” and 0.37 for “no.” ZipPy goes in the other direction, with recall of 0.26 for “yes” and 0.15 for “no.” These differences do not justify grand sociological conclusions. They do justify a practical rule: no institution should deploy an AI detector without knowing whether its error profile shifts across the populations it evaluates.

In business terms, demographics are where reputational and legal exposure become obvious. In education, this means disciplinary processes. In hiring, it means screening and writing samples. In publishing and platform governance, it means unequal enforcement. A false accusation is not just a model error. It is a cost transferred to the person least able to inspect the model.

Grade level: polished writing is not the only human writing

Grade-level evaluation matters because many detector deployments happen in schools. If a tool is used to police student writing, it needs to recognize student writing—not only adult, edited, standardized prose.

Desklib again performs strongly across grades, with recall from 0.89 in Grade 8 to 0.93 in Grade 12, and an unusually high 0.99 for Grade 9. Radar is moderate to strong, with recall from 0.57 to 0.75. E5-small remains low, generally around 0.18–0.27.

ZipPy is the warning label.

For Grade 8, ZipPy recall is 0.34, but for Grades 10, 11, and 12, recall falls to 0.02, 0.03, and 0.02. Grade 9 is different, with recall of 0.55, but that subgroup contains only 52 samples, so it should not be overread.

The likely mechanism is architectural. ZipPy relies on compression-based signals. The paper notes that compression estimates are sensitive to input length and formatting because shorter or structurally irregular texts provide fewer reliable repeated patterns. In plain English: a fast statistical trick may be fast because it avoids the expensive part where understanding lives.

For schools, this matters more than a generic detector benchmark. A detector that fails on student writing should not be treated as a student-discipline tool. At most, it might become a low-confidence signal for further review. Even that should come with subgroup testing, threshold calibration, and appeal procedures. A detector score without procedural safeguards is not evidence. It is paperwork with a confidence costume.

Age: the benchmark shows calibration drift, not a neat age story

The age category uses the Blog Authorship Corpus, grouped into teens, 20s, 30s, and 40s. This category is useful because it shifts the benchmark away from school essays and into more natural online writing.

The results do not produce a simple “older writers are penalized” or “younger writers are penalized” story. They show detector-specific calibration drift.

Desklib recall declines from 0.92 for teens to 0.80 for writers in their 40s. E5-small drops more sharply, from 0.55 for teens to 0.28 for writers in their 40s. Radar stays weak, with recall between 0.22 and 0.31. ZipPy has very high recall across all age groups, around 0.95–0.97, but precision stays close to 0.49–0.50.

That last pattern matters. High recall alone can look comforting until paired with weak precision. In an operational system, one metric rarely tells the full harm story. A detector can recognize many human texts correctly in one setup and still produce noisy decisions under another. The cost depends on how the score is used: warning label, moderation queue, disciplinary trigger, hiring screen, or compliance audit.

The age category is therefore less about age as identity and more about writing ecology. Blogs vary in style, length, topic, and self-presentation. The detector is not only reading age. It is reading the traces of genre and platform. That is exactly why aggregate “human versus AI” evaluation is too thin for real deployment.

Dialect: the detector starts grading what English is supposed to sound like

Dialect is where the paper becomes hardest to ignore.

BAID evaluates African American Vernacular English, Singlish, and Standard American English. Here, the neural detectors that looked stable on standardized writing degrade sharply.

For Desklib, recall on human-written dialect data is 0.26 for Singlish, 0.20 for AAVE, and 0.35 for Standard American English. E5-small shows a different pattern: 0.35 for Singlish, 0.71 for AAVE, and 0.97 for Standard American English. Radar reports 0.25 for Singlish, 0.52 for AAVE, and 0.72 for Standard American English. ZipPy reports very high recall across all three, from 0.98 to 0.99, but with precision around 0.49–0.50.

This is where the misconception “bias is mainly about English learners” breaks. Dialect is not broken English. Informal English is not defective English. Singlish and AAVE are not failed attempts at Standard American English. They are legitimate linguistic systems with their own syntax, pragmatics, and community context.

A detector that treats dialectal variation as suspicious is not merely making a technical mistake. It is encoding a narrow theory of legitimate writing. That theory may be invisible during aggregate testing because standardized datasets reward standard-like writing. Once dialect enters, the detector’s hidden assumptions become measurable.

For businesses, this category has direct consequences. Platforms moderating user-generated content, employers screening writing samples, customer-support teams analyzing messages, and schools evaluating student work all encounter dialectal variation. If the detector has not been audited on the language varieties present in the user base, it is not production-ready. It is a prototype with institutional consequences.

The important nuance: BAID does not prove that every detector is biased against every dialect in the same direction. The patterns differ by detector. That is precisely why the benchmark is useful. Bias auditing should not begin with moral certainty. It should begin with disaggregated measurement.

Formality: GenZ English breaks the tidy human-writing stereotype

The formality category compares GenZ-style informal English with standard English. This is the most obvious stress test for detectors trained on cleaner prose.

The results are severe. On GenZ English, Desklib recall is 0.12, E5-small recall is 0.04, and Radar recall is 0.04. Their F1 scores are also extremely low: 0.14, 0.04, and 0.02. For standard English, the corresponding recall scores improve to 0.41, 0.55, and 0.30, though those are still not exactly a victory parade.

ZipPy behaves differently. It reports 0.99 recall on GenZ English and 0.97 on standard English, with F1 scores of 0.67 and 0.70. That does not mean ZipPy has solved informal-language fairness. Its broader results remain unstable, and the paper specifically notes its sensitivity to length and formatting. But the contrast shows why detector architecture matters: different assumptions fail under different writing conditions.

GenZ English is a useful category because it is not just a demographic proxy. It is a register: short, playful, slang-heavy, compressed, context-dependent, and often intentionally non-standard. These are exactly the properties that can confuse systems trained to associate “human writing” with polished essay-like prose.

The business implication is uncomfortable but simple. If an organization uses AI detection on chat messages, social posts, comments, short reviews, student discussion posts, or customer messages, formal-document benchmarks are not enough. Informal language is not edge-case noise. It is where much of the internet lives.

Topic and political leaning: content domain still changes detector behavior

Topic-level results are less dramatic than dialect and formality, but they matter for governance.

Across topics such as arts, education, engineering, law, non-profit, student, technology, and others, Desklib generally performs well, with recall around 0.85–0.90 and F1 around 0.76–0.83. E5-small is weaker, with recall ranging roughly from 0.32 to 0.51. Radar sits lower, mostly around 0.22–0.29. ZipPy again reports high recall, mostly 0.95–0.98, with F1 around 0.65–0.67.

This category shows why procurement teams should not ask only “Which detector is best?” They should ask “Best for which content?” A detector used in legal writing, engineering documentation, student essays, and social-media moderation faces different distributions. A single threshold across all of them is operational laziness with a model attached.

Political leaning produces a different pattern. Desklib performs strongly across left, neutral, and right-leaning texts, with recall from 0.89 to 0.95. Radar reports very high recall at 0.99 across all three, with F1 around 0.68. ZipPy recall is also high, around 0.81–0.83. E5-small, however, nearly collapses: recall is 0.06 for left-leaning text, 0.03 for neutral text, and 0.08 for right-leaning text.

The political result should not be misread as evidence of a strong left-versus-right asymmetry. The paper’s more defensible point is detector-specific failure on political-register text. For any organization handling political content—media platforms, civic-tech vendors, compliance teams, campaign-monitoring tools—that distinction matters. The harm is not necessarily partisan bias in the simple cable-news sense. The harm may be that a detector fails badly on a register where accusations of manipulation are already socially explosive.

That is enough reason to audit.

The appendix tests calibration, not a second fairness thesis

The appendix evaluates detector performance on AI-generated versions of the BAID samples. This part of the paper is easy to misuse.

The authors are careful: AI-generated subgroup-conditioned samples do not represent authentic demographic or experiential identity. They are generated through prompts that simulate styles. Therefore, subgroup-level patterns on synthetic text should be treated as calibration and robustness evidence, not proof of fairness toward real groups.

That distinction is not academic hair-splitting. It separates two different questions:

The synthetic results are still informative. Desklib maintains high precision and recall across many AI-generated subgroups, often with recall above 0.97 and F1 above 0.9. E5-small also shows very high recall but lower precision, around 0.55–0.60 in many cases. Radar is uneven, especially on stylistic categories. ZipPy becomes unpredictable, often showing very low recall on shorter or more informal generated texts, including near-zero results for dialect and GenZ categories in the appendix table.

The interpretation is not “synthetic results contradict the human results.” It is more precise: detectors may recognize machine fingerprints under some controlled generations while still failing to recognize real human diversity. Those are different capabilities.

A detector can be good at catching AI text and still unfair toward humans. That sentence should probably be printed on every AI-detection vendor slide, preferably before the pricing page.

What BAID means for business users of AI detection

BAID does not say that AI detection is useless. It says that detector deployment without subgroup auditing is unserious.

For schools, the lesson is disciplinary. A detector should not be treated as a primary evidence source unless its false-accusation risk has been tested on the actual student population, including English learners, dialect speakers, grade levels, and informal writing contexts.

For HR teams, the lesson is procedural. If writing samples are screened by AI detectors, the organization needs documented subgroup evaluation and a human review process. Otherwise, the detector becomes a hidden employment filter.

For publishers and media platforms, the lesson is editorial. AI detection may support triage, but it should not automatically determine authenticity, especially for political, dialectal, or short-form content.

For compliance and procurement teams, the lesson is contractual. Vendors should not be allowed to sell aggregate scores as fairness evidence. The buyer should ask for subgroup-level performance, threshold behavior, calibration options, and failure cases by content type.

A practical evaluation checklist would look like this:

The ROI argument is also different from the vendor brochure version. The value of BAID is not that it helps organizations buy a “fair detector.” That phrase should make everyone nervous. The value is cheaper diagnosis: identifying where a detector is unreliable before it becomes a governance incident, lawsuit, disciplinary scandal, or viral screenshot.

The boundaries are narrow enough to respect and broad enough to matter

BAID has limits.

It evaluates four detectors, not the entire detector market. It focuses on English text, not multilingual or cross-lingual detection. It uses existing datasets, each with its own sampling structure and representational constraints. Some subgroups are much smaller than others, which makes certain comparisons less stable. The detectors are tested as black boxes using default thresholds, which approximates casual deployment but does not reveal how performance might change after careful calibration.

There is also a metrics interpretation issue worth handling carefully. The paper’s human-written evaluation uses precision, recall, and F1 by subgroup, but for practical fairness interpretation, recall on human-written text is the most intuitive false-accusation proxy: low recall means real human writing is not being recognized as human. Precision and F1 are still useful, but they should be read within the authors’ evaluation setup rather than treated as universal deployment rates.

These limitations do not weaken the paper’s core message. They define its proper use. BAID is not a certification system. It is a diagnostic benchmark. It does not tell an organization, “This detector is safe.” It tells the organization where safety claims begin to fall apart.

That is already valuable.

The detector is not neutral just because it is statistical

The temptation with AI detectors is to treat them as neutral referees. They produce numbers. Numbers feel less biased than people. Unfortunately, a number can be biased with excellent formatting.

BAID’s main contribution is to make that bias inspectable. It shows that the same detector can behave differently across demographics, grade levels, age groups, dialects, formality registers, topics, and political writing. It also shows that architecture matters: neural detectors, adversarial detectors, lightweight encoders, and compression-based systems fail in different ways.

The practical conclusion is not dramatic. It is administrative, which is where many AI harms actually live.

Do not deploy AI-text detectors as punitive tools without subgroup audits. Do not accept aggregate F1 as proof of fairness. Do not treat synthetic style rewrites as demographic evidence. Do not use one threshold across every writing context. And please, for the sake of everyone who has ever written a sentence that did not sound like a polished corporate memo, do not confuse standard English with human authenticity.

AI detection may still have a role. But after BAID, that role has to be narrower, audited, and procedurally humble.

A detector that cannot recognize how real people write should not be trusted to decide who wrote what.

Cognaptus: Automate the Present, Incubate the Future.