AI Risk

TL;DR for operators Most model-governance systems still treat LLM failure like a customer-support ticket: hallucination, bias, unsafe compliance, sycophancy, escalation, add a dashboard, summon a committee, repeat until morale improves. NeuroCogMap proposes a more useful question: when the model fails, which internal systems were recruited, under-recruited, or misrouted? The paper builds a functional atlas of LLM internals by clustering sparse autoencoder features into parcels, attaching cognitive descriptions to those parcels, mapping them to capabilities, and arranging those capabilities into a four-level hierarchy: perception, representation, abstraction, and application.1 ...

AI agents are excellent at finding the obvious answer. That sounds like a compliment until the task is to avoid everyone else’s obvious answer. Imagine three firms using AI assistants to screen applicants, forecast demand, or decide which customer segments deserve attention. If the goal is consistency, shared focal points are useful. Everyone reads the same policy, applies similar criteria, and avoids the usual mess of human improvisation. Lovely. The spreadsheet smiles. ...

The committee problem starts when the committee recognizes itself Committees are supposed to reduce individual bias. Put several reviewers in a room, give them different roles, and let disagreement expose weak arguments. This is the polite theory of institutional decision-making. It is also the theory behind many multi-agent AI pipelines. A critical model reviews the claim. A balanced model moderates the tone. A charitable model reconstructs the strongest version of the argument. A supervisor aggregates the outputs. Somewhere nearby, a fact-checking layer pulls external evidence. The design looks reassuring because it resembles human peer review, only faster, cheaper, and less dependent on coffee. ...

Scorecards look objective until a user reads the explanation Scorecards are comforting. They turn a messy judgment into a neat row of numbers: sparsity, proximity, plausibility, trust score, completeness. The model team can rank explanation methods. The governance team can file the validation report. The product team can say the system is explainable. Everyone gets to leave the meeting before dinner. ...

A dashboard says the customer-service agent resolved the ticket. The log says it retrieved the policy document, summarized the complaint, checked the refund rule, and sent a polite reply. The manager sees the outcome and asks the obvious question: why did the system approve the refund? For a normal machine-learning model, this question has a familiar shape. Which features mattered? Which tokens were important? Which image region pushed the classifier toward one label? We have a whole shelf of explainability tools for that shelf-sized problem. ...

Taxi dispatch is not a poetry recital. When a passenger calls and says, “I’m on Arguello,” the system does not need to appreciate the full expressive richness of the sentence. It needs to identify one street name, map it to the right place, and send a vehicle there. This is not a broad language-understanding task. It is a narrow operational task with coordinates attached. ...

Explanations should answer the question people actually ask Audit meeting. A model has made a decision. Someone projects a heatmap. The highlighted pixels are around a chin, an eye, a forehead, or some other facial region that looks important because the model says it is important. Everyone nods carefully. Nobody is much wiser. The model has technically been “explained,” in the same way a smoke alarm explains fire by making noise. ...

TL;DR for operators Training data does not become risky only after a model has memorised it. It often leaves signals while training is still happening. That is the useful idea behind Generative Data Cartography, or GenDataCarto: track how each pretraining sample behaves during early training, then use that behaviour to decide which data should be kept, up-sampled, down-weighted, or removed.1 The method uses two signals. The first is early loss, which approximates how difficult a sample is. The second is the frequency of “forget events”, where a sample appears learned and later becomes poorly fitted again. In the paper’s framing, frequent forget events are not just training noise. They are a warning that a sample may be unusually influential, repeatedly re-entering the model’s attention like a guest who refuses to leave the meeting. ...

TL;DR for operators A bad agent incident rarely starts with one dramatic mistake. It usually forms as a chain. The system may be predisposed to fail because of training data, feedback, system prompts, or scaffolding. The environment may then trigger the failure through unclear tasks, insecure information, unavailable tools, excessive permissions, or malicious inputs. Finally, the agent may commit a visible cognitive error: it overlooks something, misunderstands a command, chooses the wrong goal, or executes an action badly. ...

TL;DR for operators A model that says “I don’t know” is not automatically trustworthy. It may be cautious. It may be badly calibrated. It may be uncertain for the wrong reasons. It may also be using uncertainty as a very elegant trapdoor. Polite refusal, unfortunately, is still refusal. Stephan Rabanser’s thesis, Uncertainty-Driven Reliability: Selective Prediction and Trustworthy Deployment in Modern Machine Learning, is useful because it treats uncertainty not as a philosophical mood, but as an operational control layer.1 The key question is not whether a model can emit a confidence score. Most models can emit something confidence-shaped. The harder question is whether that score can decide which cases should be automated, deferred, reviewed, rejected, routed to a larger model, or audited. ...