A Different AI Writer, a Different Shortlist

TL;DR for operators

An AI recruitment screener can change which applicants advance even when their CVs and job descriptions remain unchanged. The model used to write each cover letter can influence the evaluator’s scores and candidate rankings.

In a controlled study by Pagan and colleagues,¹ the most adverse pooled writer pairing produced 7.3 additional percentage points of simulated shortlist displacement, equivalent to approximately 1.8 of 25 initially higher-ranked candidates. This was displacement beyond a same-writer baseline, not an observed hiring-error rate.

For recruitment operators, the practical response is to audit the actual screening workflow: hold candidate qualifications constant, vary the cover-letter writer, and measure changes in scores and shortlist membership. Supplying the CV reduces average writer preferences, but does not eliminate them. Evaluator selection should therefore consider ranking stability and writer sensitivity, not general model performance alone.

The Writing Model Can Change Who Advances

Consider a recruitment team that receives 50 applications for a position and uses an AI evaluator to select 25 candidates for further consideration. Applicants submit CVs and cover letters, some prepared with different AI writing assistants.

The screening system uses the same scoring rubric for everyone. Yet standardizing the evaluator does not necessarily make its judgments independent of the writing tools applicants use. Two letters presenting the same qualifications may receive systematically different assessments because they were generated by different models.

The consequence becomes measurable when candidate rankings change. An initially stronger applicant can fall below the shortlisting threshold while a lower-ranked applicant moves above it, without any change to their underlying CVs.

Pagan and colleagues investigate this possibility through a controlled computational audit. Their design isolates the influence of the writing model while keeping candidate and job information fixed. The study examines whether these preferences exist, whether they affect simulated shortlisting, and which evaluator characteristics are associated with greater sensitivity.

Holding Qualifications Fixed Reveals a Writer Hierarchy

The researchers pair 50 candidate CVs with each of ten job descriptions. Eleven AI models independently generate cover letters for those candidate-job combinations, producing 5,500 letters.

Nine API-hosted LLM evaluators then score applications under three conditions: CV only, cover letter only, and CV plus cover letter. Four independent scoring calls per combination yield 414,000 evaluations.

The essential control is that each evaluator can assess the same candidate applying for the same job using letters written by different models. Differences in the resulting assessments cannot be attributed to changing the candidate’s CV or the job description.

To measure preference, the researchers compare each writer’s score with the evaluator’s average score across all writers for that same candidate and job. The resulting Relative Preference Gap measures systematic preference or disfavor relative to this within-candidate baseline.

In cover-letter-only evaluation, Gemini 3.5 Flash achieves an average gap of +0.45 points, GPT-4o-mini +0.44, and Gemini 3 Flash +0.36. Llama-3.1-8B records -0.53.

These differences establish a generative preference hierarchy: particular writing models are systematically favored by the audited evaluators. They do not establish that the highest-ranked models produce objectively better prose or more accurate representations of candidate qualifications.

The researchers also verify that their constructed candidate groups contain meaningful differences in apparent fit. Candidates designated high-fit through CV-to-job embedding similarity receive an average CV-only score of 4.20, compared with 1.95 for moderate-fit candidates. Evaluators’ independent CV-only top-half selections agree with the embedding-based tiers for 86.0%–93.6% of candidates (Appendix A.5, Table A2). This supports the experimental grouping, although it is not independent human validation of applicant quality.

A Score Preference Becomes a Shortlisting Consequence

Score differences are operationally consequential when they change which candidates cross a selection threshold.

The researchers simulate this by defining the top 25 candidates for each evaluator and job using that evaluator’s CV-only rankings. They then assign one writing model to the initially higher-ranked candidates and another to the lower-ranked candidates.

Some candidates change position even when both groups use the same writing model. Across same-writer conditions, displacement ranges from 7.6% to 9.5%, averaging 8.2%.

The paper therefore calculates net rank displacement by subtracting same-writer displacement from displacement under different writer assignments. This isolates the additional ranking change associated with the mismatch between writing models within the simulation.

The most adverse pooled pairing assigns Llama-3.1-8B letters to initially higher-ranked candidates and DeepSeek V3.2 letters to lower-ranked candidates. The result is +7.3 percentage points of net displacement, with a cross-evaluator standard deviation of 3.2 percentage points (Section 2.4, Figure 4).

That increment corresponds to approximately 1.8 of the 25 initially higher-ranked candidates. Particular evaluator-writer combinations produce net displacement reaching 12.8 percentage points (Appendix B).

The 7.3-point result must not be read as a 7.3% probability that a real applicant will be wrongly rejected. It is an additional displacement rate measured against a same-writer control in a constructed 50-candidate shortlist. The initially higher-ranked group is defined by AI-based CV assessment, not independently verified human hiring judgments.

Nevertheless, the experiment identifies a concrete failure mode for automated screening: presentation differences can alter selection outcomes even when the underlying candidate evidence remains fixed.

Providing the CV Helps, but Changes Which Evaluator Is Most Neutral

A recruitment operator might expect that providing the CV alongside the cover letter would largely resolve presentation-driven scoring differences.

The experiment finds substantial attenuation, but not elimination.

The smaller preference gaps indicate that substantive CV information reduces the influence of writer-dependent presentation. The decline in ranking agreement also indicates that evaluators become less consistent with one another about which writing models they favor.

Individual writer positions can change markedly. GPT-4o-mini moves from a +0.44 average gap in cover-letter-only evaluation to -0.05 when the CV is included.

Evaluator neutrality is similarly conditional. The paper measures systematic writer sensitivity through cumulative absolute rank deviations, where lower values indicate less favoritism. Claude Sonnet 4.6 performs best under cover-letter-only scoring, while Gemini 3.5 Flash leads under combined evaluation. DeepSeek V3.2 has the largest combined-condition rank-gap measure, approximately 14.9.

This comparison is main evidence of workflow-dependent evaluator behavior, not a recommendation to universally adopt either leading model. A recruitment team needs to test the evaluator with the same documents, prompts, and shortlisting rules it intends to deploy.

Observable Writing Features Explain Much of the Preference Pattern

The researchers also investigate why writing models receive different evaluations.

They measure textual properties including readability, structure, vocabulary, emotion, and semantic similarity to the advertised job. Per-evaluator Ridge regressions then associate these features with evaluation scores.

In cover-letter-only assessment, similarity between the letter and the job advertisement has the strongest and most consistently positive association, with a mean standardized coefficient near +0.83. Flesch Reading Ease follows at approximately +0.30.

When the CV is included, similarity to the job advertisement and similarity to the candidate CV each have mean coefficients near +0.60.

These are conditional statistical associations. They do not establish that mechanically repeating job-advertisement language or changing an isolated writing feature would causally improve candidate evaluation.

A supplementary Shapley variance decomposition provides another perspective. In combined evaluation, job, candidate-substance, and evaluator baseline predictors explain approximately 90.6% of pooled score variance. Adding categorical writer identity raises explained variance to about 91.1%; adding measured text features instead raises it to approximately 91.0% (Appendix C.3).

Measured textual characteristics therefore capture nearly as much explanatory variation as writer identity under this specification. The result supports investigating observable features, although correlated predictors and differences in evaluator score dispersion limit mechanistic interpretation.

It also does not establish ranking fairness. The appendix finds that raw-score variance measures and rank-based bias measures can diverge. A feature model that explains scores well is not necessarily sufficient to predict shortlist displacement.

The two locally hosted writers, Llama-3.1-8B and DeepSeek-R1-8B, also fare worse than the stronger API-hosted writers in the audited pool. In combined-condition head-to-head comparisons, their average win rates are 36% and 42%, compared with 56% for Gemini 3.5 Flash and GPT-5-mini.

However, both locally hosted models are approximately 8B parameters, while the API-hosted models differ in capacity and capability. The experiment cannot attribute their disadvantage specifically to local hosting. It establishes a model-pool difference, not an inherent penalty against locally deployed AI.

What Recruitment Operators Can Audit

The paper’s most durable contribution is the auditing design, which can be repeated as writing and evaluation models change.

For a recruitment team selecting an AI screening evaluator, Cognaptus draws four practical implications:

1. Build fixed-candidate comparisons. Generate alternative cover letters using different writing models while holding CVs, job descriptions, and evaluation rubrics constant. Compare the evaluator’s treatment of the same applicant across variants.

2. Test the actual document workflow. Evaluate cover-letter-only and CV-plus-cover-letter conditions to determine how access to substantive applicant information changes writer sensitivity.

3. Measure shortlist movement against controls. Establish CV-only rankings, compare writer assignments, and subtract same-writer displacement. Pay particular attention to applicants near the intended selection threshold.

4. Repeat audits after model changes. Monitor rank gaps, displacement, and observable writing-feature associations when evaluator versions, writer models, or prompts change. Use repeated evaluations to assess scoring stability.

These are proposed operational applications of the experimental design, not interventions whose effectiveness has been established in production hiring.

The Boundary Is Simulated Screening, Not Observed Hiring

Several conditions restrict generalization.

The candidate-job pairings were constructed by researchers from public datasets, not drawn from naturally occurring applicant pools. The experiment covers ten selected jobs, standardized English-language letters, and nine API-hosted evaluator models. High-fit status is determined through embeddings or evaluator-specific CV-only rankings rather than independently validated recruiter assessments.

The study observes neither interviews nor actual employer hiring decisions. It also cannot determine whether writer sensitivity interacts with protected demographic characteristics or causes socioeconomic disadvantage through unequal access to writing tools.

Those limitations do not undermine the controlled finding that changing the cover-letter writer can change AI evaluator scores and simulated rankings. They define what remains to be established before drawing conclusions about real-world discrimination or the effectiveness of specific safeguards.

For recruitment operators, the immediate lesson is methodological: an AI evaluator should be tested for sensitivity to how qualifications are presented, not only for how accurately it appears to assess the qualifications themselves. The relevant outcome is whether those sensitivities alter the candidate decisions the system is expected to support.

Cognaptus: Automate the Present, Incubate the Future.