Cover image

When the Rule Changes but the Verdict Doesn't

TL;DR for operators An AI compliance checker can classify more than 90% of cases correctly while leaving most verdicts unchanged when the supplied regulation is removed or modified. In a five-model audit spanning 20 regulatory domains, the four general-purpose models achieved 90.3–91.9% accuracy but only 7.0–9.4% aggregate sensitivity to rule changes. For teams procuring or deploying compliance automation, benchmark accuracy does not establish that the system’s decisions depend on the customer’s governing rules. Counterfactual tests that delete, substitute, or negate rules provide an additional diagnostic. But low sensitivity is not proof that a model ignored the regulation: some scenarios independently reveal the likely verdict, and sensitivity changes substantially across datasets. ...

November 14, 2026 · 8 min · Zelina
Cover image

What 99% Review Coverage Fails to Measure

TL;DR for operators A system can produce structured records for almost every customer review while missing much of the information those records are supposed to capture. In a study of Pashupatinath Temple reviews, a culturally adapted GPT-4o pipeline generated entity outputs for 99.4% of 2,500 reviews, yet achieved only 28.4% provisional entity micro-F1, with 21.8% recall, against a human-adjudicated reference. Its aspect-sentiment results showed a similar gap: 79.4% polarity agreement when both the model and reference detected an aspect, but only 42.6% F1 when correct aspect detection and sentiment classification were both required. ...

November 13, 2026 · 8 min · Zelina
Cover image

When Helpful Examples Still Lead to Refusal

TL;DR for operators Correctly labeled helpful examples do not necessarily eliminate false refusals in safety-aligned language models. Training updates from harmful requests can interfere with learning to answer legitimate requests that contain sensitive language. Chen and colleagues introduce GOAL, a fine-tuning method that selectively removes conflicting gradient components from helpful-response training and separately encourages the model to distinguish harmful from harmless-but-sensitive prompts.1 Unlike simply reducing safety supervision, the method leaves harmful-example training gradients intact. ...

November 9, 2026 · 8 min · Zelina
Cover image

A Novel Hypothesis May Still Follow an Old Direction

TL;DR for operators An AI research assistant may produce proposals that look substantially different from existing papers without identifying fundamentally new research directions. A study comparing 524,308 LLM-generated hypotheses with published scientific research finds that generated ideas have 15–23% lower median directional novelty, even though most evaluated models produce hypotheses farther from their nearest published neighbors. ...

November 4, 2026 · 8 min · Zelina
Cover image

Code Translation Is Not a Round Trip

TL;DR for operators An analytical workflow can be translated into code that executes successfully while silently changing its results. On ORCA-Main, Claude-Opus-4.6 achieves 75.07% success translating NumPy to Pandas but only 32.93% in the reverse direction—a 42.14-percentage-point gap. The ORCA benchmark1 evaluates this problem across 1,800 data science translation tasks. The asymmetry challenges the assumption that migration is primarily a matter of replacing equivalent API calls. Translation difficulty also depends on how source-library operations can be reconstructed in the target library. Complete workflows introduce another risk: individually correct transformations can still produce incompatible schemas, feature ordering, or preprocessing states. The leading model achieves 56.92% on the full ORCA-Main benchmark and 33.67% on ORCA-Project under a separate evaluation configuration; these scores are not a controlled comparison of project difficulty. ...

October 26, 2026 · 8 min · Zelina
Cover image

Relevance Is Not Verification: Choosing an ESG RAG Model

TL;DR for operators A sustainability analytics team can receive an AI-generated answer that addresses its question and follows a retrieved corporate disclosure, yet still lacks the information needed for accurate reporting. A benchmark of seven open-source language models for ESG question answering confirms that these qualities must be evaluated separately.1 Answer relevancy ranged from 0.760 to 0.881, while factual correctness remained between 0.387 and 0.449. Different models led the three principal generation metrics. ...

October 26, 2026 · 8 min · Zelina
Cover image

When Confidence Tracks Consistency Instead of Correctness

TL;DR for operators An AI assistant’s confidence score can become substantially better at predicting correctness without becoming reliable at recognizing its own mistakes. Across 10 open-weight LLMs and five dataset variants, the mean correlation between confidence and the probability assigned to the correct answer rose from 0.04 to 0.73. However, the comparison also changed confidence measurement from verbal reporting to a learned probe, so the improvement cannot be attributed to training alone. ...

October 26, 2026 · 8 min · Zelina
Cover image

When Helpful Starts Deciding for You

TL;DR for operators For subjective personal questions, the main risk is not limited to whether an AI gives a wrong answer. It is also whether the system helps a user form a judgment or supplies the judgment for them. Cheng and colleagues call the latter pattern LLM-as-oracle use.1 Their evidence spans public conversation logs, donated longitudinal histories, a preregistered experiment, a randomized three-week interaction study, and response-policy interventions. ...

October 14, 2026 · 7 min · Zelina
Cover image

Split the Catalog, Not the Hierarchy

TL;DR for operators Large product catalogs, document collections, and knowledge inventories eventually reach a scale where asking one model to reason over every label at once becomes difficult. Splitting the labels into smaller groups solves part of the input problem, but it creates another one: the pieces still have to reconnect into a coherent hierarchy. ...

October 10, 2026 · 8 min · Zelina
Cover image

Check What Breaks: Turning Reasoning Errors into Inference-Time Controls

TL;DR for operators A generic instruction such as “pay more attention” is not a reliable reasoning control. In this study, that prompt barely changes performance for one model, substantially improves another, and makes a third slightly worse. A prompt that instead tells the model which failure points to verify reduces incorrect answers across all three tested model families. ...

October 2, 2026 · 7 min · Zelina