Peer Review in the Age of Agents: When Scientists Go Silicon

Reviewers are the unglamorous load-bearing wall of science. They slow things down, miss things, disagree with each other, and occasionally write comments that make authors reconsider their life choices. They are also the reason published knowledge is not just a PDF-shaped rumour.

So when a conference lets AI agents act as both primary authors and reviewers, the tempting story writes itself: silicon scientists have entered the building, peer review is next, and human academics can finally retire into committee work, where they have been spiritually living for years.

That is not what the paper shows.

The paper on Agents4Science reports something more useful and more awkward: AI can now participate seriously in scientific production, but the quality signal improves when humans remain deeply involved in problem framing, experimental design, verification, and judgment.¹ The interesting result is not that AI wrote papers. The interesting result is where the system needed disclosure, calibration, adversarial checks, reference verification, and human review to stop the whole thing turning into a very polished guessing machine.

For business leaders, that matters because research is only one version of a broader knowledge-work problem. The same pattern appears in technical due diligence, investment memos, compliance reviews, market intelligence, product research, clinical documentation, engineering analysis, and board reporting. AI can produce and critique work at scale. The question is whether the organisation has built the machinery to tell useful work from confident nonsense.

Agents4Science is less a prophecy of autonomous science than a field test of AI governance. Conveniently, it came with receipts.

The useful evidence is not the conference gimmick

Agents4Science was designed as a live experiment in AI-led science. Submissions were expected to list an AI model as the primary author, with humans as co-authors. Each submission had to complete two checklists: one adapted from NeurIPS-style methodological and ethical standards, and another specific to AI involvement across the research process.

The scale was large enough to be informative without pretending to settle the future of science. The conference received 315 submissions. Sixty-two were incomplete and desk-rejected, leaving 253 complete submissions. Those complete papers were reviewed by three LLM reviewers: GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4. The 79 top-scoring papers then received human expert assessment. The programme committee accepted 48 papers after considering both AI and human reviews.

That design matters. This was not a clean laboratory benchmark in which one model answers a fixed set of questions and gets a leaderboard score. It was a messy, operational setting: authors used different models, domains varied, reviewers disagreed, references needed checking, and some submissions tried to manipulate the review system. In other words, it resembled reality. Terrible for neat conclusions; excellent for useful ones.

The paper’s evidence falls into four buckets:

That last distinction matters. A weak reading would say, “AI scientists are here.” A stronger reading says, “AI scientific workflows are becoming possible only when surrounded by inspection points.” Less cinematic, more useful.

Accepted papers leaned toward collaboration, not total autonomy

The conference asked authors to disclose AI involvement across four stages: hypothesis development, experimental design and implementation, data analysis and interpretation, and manuscript writing. The disclosure categories ranged from Category A, meaning at least 95% human contribution, to Category D, meaning at least 95% AI contribution. The middle categories captured mixed human-led or AI-led work.

The headline figure is seductive: among all submissions, fully AI-driven research across all four stages was the most common pattern, accounting for 23.3% of submissions. Over half of both all submissions and accepted papers reported primary AI contribution, Category C or D, in every stage of the research process.

But the acceptance pattern complicates the “AI can do it alone” story. The share of fully AI-driven papers dropped from 23.3% among submissions to 14.9% among accepted papers. The authors also observed a broader shift toward more human-AI collaboration among accepted submissions. Human involvement was especially higher in the early stages: hypothesis development and experimental design. AI autonomy was stronger later, particularly in data analysis and manuscript writing.

This is the paper’s most business-relevant signal.

In research, as in most high-value knowledge work, the hard part is often not generating output. It is deciding what question is worth asking, what assumptions deserve stress-testing, what experiment would actually distinguish between explanations, and what result should change a decision. AI is already strong at downstream production: drafting, coding, summarising, formatting, recombining, checking, and producing variations at industrial speed. But the accepted papers suggest that quality still benefits from human control over the upstream frame.

That is not a sentimental claim about human uniqueness. It is an operational claim about error placement. If AI makes a mistake in wording, a reviewer may catch it. If AI makes a mistake in the research question, the whole project can be elegantly misdirected. The earlier the error, the more expensive the fluency.

This should sound familiar to any company deploying AI agents internally. The same division of labour applies to market analysis, legal review, procurement analysis, financial modelling, product strategy, and technical evaluation.

Delegate execution aggressively. Do not delegate intention casually.

The reviewer models were useful, but not interchangeable

The conference did not merely let AI produce papers. It also used LLMs to review them.

Each complete submission was reviewed by GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4 using NeurIPS 2025 reviewing guidelines. Scores ran from 1 to 6, with higher scores more positive. Papers with an average LLM score of at least 4.0 advanced to human review.

The reviewers were calibrated using anonymised ICLR 2022 and ICLR 2025 papers, along with human acceptance decisions and review scores. This calibration step is not a decorative implementation detail. It is the difference between “we asked a chatbot whether it liked the paper” and “we tried to align model scoring with an existing review regime.” The distinction is not small.

Even after calibration, the models behaved differently. Their pairwise score correlations were positive, with an average Pearson correlation of 0.48, but their scoring tendencies diverged sharply. GPT-5 was the harshest reviewer, with an average score of 2.30. Gemini 2.5 Pro was the most positive, with an average score of 4.23. Claude Sonnet 4 sat between them at 3.0.

Against the subset of 79 papers also reviewed by human experts, GPT-5 and Claude Sonnet 4 were closer to human scores, with mean absolute differences of 0.91 and 1.09 respectively. Gemini 2.5 Pro was much further away, with a mean absolute difference of 2.73.

There are two lessons here, and neither is “pick the best model and relax.”

First, LLM reviewers can catch concrete issues. The paper gives examples where reviewers identified limited scope, mismatches between abstracts and body text, contradictory claims, and numerical discrepancies such as inconsistent reported $R^2$ values. This is exactly the kind of review labour that scales well: checking internal consistency, looking for missing caveats, comparing claims across sections, and flagging suspicious numbers.

Second, reviewer models have temperament. Gemini 2.5 Pro, in the authors’ examples, showed sycophancy: praising work as technically flawless and highly impactful where a human reviewer was more sceptical about incomplete experiments and selective presentation. That is not just a personality quirk. In a review pipeline, inflated positivity changes triage. Harshness changes triage too. A model’s scoring style becomes part of the institution’s decision system.

Businesses should pay attention. When AI agents review contracts, investment theses, compliance files, vendor proposals, incident reports, medical summaries, or engineering plans, they are not neutral machines dispensing pure judgment. They encode thresholds. Some will over-approve. Some will over-reject. Some will catch arithmetic but miss strategic nonsense. Some will sound balanced while being wrong in a very reassuring voice, the most dangerous voice in enterprise software.

The practical answer is not to anthropomorphise model “personalities.” It is to instrument them. Track score distributions. Compare against human decisions. Calibrate prompts and rubrics. Use model panels where disagreement is informative. Escalate high-impact cases to humans. Treat reviewer models as measurement tools with biases, not as tiny professors trapped in an API.

The integrity checks were the real governance layer

The paper’s most sobering evidence does not come from the accepted papers. It comes from the checks around them.

The organisers built an automated reference verification system. For each submission, the system extracted reference titles and available metadata, searched the web for matches, and flagged references where no match was found. The method is not a perfect oracle. A reference could be real but hard to match. A malformed citation could be flagged even when the underlying work exists. But as a screening tool, it exposes the right operational problem.

Only approximately 44% of submissions — 111 papers — had no hallucinated references detected. The remaining papers had at least one reference flagged as problematic.

That is not a minor formatting issue. In research, references are part of the evidence chain. In business, their equivalents are source documents, contract clauses, transaction records, market data, regulatory citations, customer evidence, audit trails, and prior decisions. If an AI system fabricates or loosely matches those anchors, the output may look professional while quietly breaking provenance.

The authors also deployed a system to detect prompt injections and adversarial instructions intended to manipulate LLM reviewers. It found two papers that attempted to game the review process. Those papers were not accepted.

This is the part many AI adoption plans still treat as optional. It is not optional. Once AI systems are used to review other AI-generated work, adversarial behaviour becomes part of the environment. People will try to influence the evaluator. Documents will include hidden instructions. Vendors will optimise proposals for automated scoring. Employees will learn what wording causes a system to approve a request. The review layer becomes a target.

In that world, “human in the loop” is too vague to be a control. The loop needs architecture.

A serious AI review workflow needs at least five layers:

1. Disclosure by workflow stage: Who or what generated the hypothesis, design, analysis, interpretation, and final text?
2. Automated consistency checks: Do numbers, claims, figures, citations, and conclusions agree across the document?
3. Reference and provenance verification: Can cited evidence be matched to real, accessible sources?
4. Adversarial-content scanning: Does the submission attempt to manipulate the model reviewer?
5. Human escalation rules: Which cases require expert review, and which can be handled by automated triage?

Agents4Science did not prove that this stack is complete. It did show that without something like it, AI-led review becomes dangerously easy to fool and embarrassingly easy to flatter.

The business lesson is review architecture, not AI authorship

The obvious business takeaway is that AI can accelerate research workflows. True, but not very interesting. Speed is the least surprising thing about automation.

The sharper lesson is that AI changes where quality control must live. In traditional knowledge work, review usually happens after a human has produced a draft. In agentic workflows, review must be distributed across the whole pipeline because the system can generate, justify, cite, revise, and evaluate its own work. That creates a recursion problem. The machine is both author and critic. Delightful, in the same way a bank letting borrowers approve their own loans would be delightful.

The paper points toward a more disciplined architecture.

For enterprises, the answer is not “replace reviewers with LLMs.” It is “rebuild review as an evidence architecture.”

That architecture looks different from a chatbot interface. It includes structured intake, declared autonomy levels, source tracking, model-generated critiques, independent model disagreement, automated factual checks, adversarial scanning, and expert escalation. The user experience may still look simple. The backend should not.

This distinction matters because many organisations are currently deploying AI as if the front-end prompt is the product. It is not. The product is the governance system behind the prompt. The prompt is just the politely worded trapdoor.

AI is better downstream than upstream, and that is still valuable

The paper’s authors report that human co-authors frequently described model limitations: hallucinated or loosely related references, overclaimed results, erroneous code, context-length problems, formatting issues, and lack of creativity. Several authors noted that models struggled to generate novel or complex experimental ideas beyond given templates, or lacked deep domain expertise and nuanced interpretation.

This is not a death sentence for AI research agents. It is a deployment map.

AI appears more reliable when the task is bounded, checkable, and downstream of a clear frame. That includes drafting manuscripts, generating code, summarising literature, running analyses, checking internal consistency, and producing reviewer-style feedback. It appears riskier when asked to originate deep scientific direction, interpret ambiguous results, or judge novelty in a domain where surface fluency is cheap and real insight is expensive.

Businesses should resist the childish version of the question: “Can AI replace experts?” It is vague and usually asked by someone hoping the answer will be budget-friendly.

The better question is: “Which parts of expert work can be decomposed into bounded tasks with verifiable outputs, and which parts require accountable judgment?”

Agents4Science suggests a practical split:

• Let AI generate variants, drafts, checks, summaries, code, and review comments.
• Let humans own problem framing, evaluation criteria, final interpretation, and high-stakes approval.
• Use automated systems to verify evidence anchors before humans waste time debating polished fiction.
• Use model disagreement as a signal, not as an inconvenience.
• Keep records of model identity, prompt rubric, autonomy level, and verification outcomes.

This is not glamorous. Neither is accounting. Both exist because civilisation has learned that numbers and claims behave better when watched.

Boundaries: what this paper should not be used to claim

Agents4Science is important because it is concrete. That does not make it universal.

First, the submissions were heavily weighted toward AI and machine learning. AI and Machine Learning accounted for 64.3% of all complete submissions and 69.6% of accepted papers. That means the participant pool was likely unusually comfortable with AI tools and AI evaluation norms. A conference on ecology, law, materials science, public health, or macroeconomics might show different patterns.

Second, autonomy disclosures were self-reported. The paper’s AI-involvement checklist is valuable precisely because it makes collaboration visible, but it is not an audit log. Authors may interpret categories differently. They may overstate or understate AI contribution. The disclosure system is a governance improvement, not a measurement instrument carved into stone.

Third, human experts reviewed only the 79 top-scoring papers after LLM screening. That makes sense operationally, but it means the paper is not a full head-to-head comparison of LLM and human review across all submissions. It tells us about triage and selected human assessment, not a universal replacement test.

Fourth, the reference verification system was a screening system based on search matching. A flagged reference is a warning, not a final verdict. The finding is still serious because the warning rate was high, but businesses should not confuse automated flags with completed adjudication.

Finally, the conference setting creates incentives that may not match ordinary enterprise workflows. Authors knew AI involvement was expected. Reviewers used conference rubrics. The work was public-facing. Internal corporate research, compliance review, diligence, and technical QA will have different incentives, risk tolerances, and failure modes.

These boundaries do not weaken the paper’s practical value. They clarify it. The result is not “AI can do science alone.” The result is “AI-led knowledge production becomes useful when designed as a transparent, checked, human-governed system.”

That is a less dramatic sentence. It is also the one worth building around.

Peer review becomes a system, not a person

The future suggested by Agents4Science is not a world where reviewers disappear. It is a world where review becomes more layered.

A paper, memo, model output, or technical report may first pass through automated provenance checks. Then one model may test for internal contradictions. Another may score methodological soundness. Another may search for missing limitations. A separate system may detect adversarial instructions. Human experts may then review the subset where the stakes, uncertainty, novelty, or disagreement justify their time.

This is not the death of peer review. It is the industrialisation of pre-review.

The danger is that organisations will adopt the cheap half of this system and skip the expensive half. They will automate production and review, but not disclosure. They will deploy AI reviewers, but not calibrate them. They will demand speed, but not provenance. They will use generated citations, but not verify them. They will call the result “AI transformation,” because apparently every era needs its own euphemism for undercontrolled process risk.

The opportunity is more interesting. Companies that build review architecture well can increase throughput without surrendering judgment. They can use AI to make experts more effective rather than merely busier. They can catch low-level errors before they reach senior people. They can compare reviewer models and detect systematic bias. They can log where AI contributed and where humans intervened. They can turn knowledge work from artisanal document passing into an auditable workflow.

That is where the real business value sits. Not in silicon scientists replacing human ones, but in research operations becoming measurable, inspectable, and scalable.

Agents4Science shows that AI can write, review, flatter, hallucinate, catch errors, and sometimes help produce serious work. A charming résumé, if also a slightly alarming one. The serious conclusion is that AI agents should not be treated as autonomous colleagues or decorative assistants. They should be treated as powerful process components inside a governed system.

The winners will not be the organisations that ask AI to think harder. They will be the ones that design the review stack so thinking, checking, evidence, and accountability are no longer conveniently tangled together.

Cognaptus: Automate the Present, Incubate the Future.