Peer Pressure: AI Reviewers Pass the Item Test, Not the Replacement Test
Review is a strange business process. The visible output is a verdict: accept, reject, revise, approve, block, escalate. The useful output is usually smaller and more annoying: one specific criticism that is correct, important, and supported by evidence. That distinction is where the new paper On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists becomes more interesting than the usual “can AI replace reviewers?” theatre.1 The paper does not ask whether an AI reviewer can imitate a human reviewer’s overall score. It asks whether each individual criticism is any good. ...