Llm-Evaluation

TL;DR for operators Most AI quality programmes still treat truthfulness as a factual accuracy problem: did the model get the answer right, cite the source, or hallucinate a feature that does not exist? That is necessary. It is not sufficient. The paper behind this article argues for a nastier category: “machine bullshit,” meaning model output produced with indifference to truth rather than simple ignorance or random hallucination.1 The key point is not that models become stupid. It is that, under some incentives, their outward claims stop tracking what they appear to know. ...

TL;DR for operators Most model-selection dashboards still ask the wrong question. They ask which LLM gives the best accuracy for the lowest inference cost. Zellinger and Thomson’s paper asks a more operationally honest one: how much does a wrong answer, a slow answer, or no answer cost in this specific workflow?1 The paper’s useful move is to convert competing performance metrics into a single expected dollar reward. Inference cost stays in dollars. Latency gets priced in dollars per second or minute. Errors get priced by their business consequence. Abstention gets priced by the cost of failing to answer or escalating to a human. Once everything is in the same unit, the “best model” is no longer the one that looks attractive on a Pareto plot. It is the model with the highest expected reward under the actual economics of the task. ...

TL;DR for operators Most AI evaluation still asks whether a model can produce the right answer. This paper asks a quieter but more commercially awkward question: when a model uses a word, does it attach human-like emotional, concrete, familiar, gendered, or sensory associations to that word?1 The authors propose using established psycholinguistic word norms as an automated alignment test. Instead of hiring new human raters every time, they reuse datasets where humans have already rated thousands of English words on features such as arousal, valence, concreteness, imageability, familiarity, gender association, and sensory modalities. ...

TL;DR for operators Most AI evaluations still ask the wrongly narrow question: did the model get the answer right? That is useful, but it is not enough when the model is expected to act as an agent, revise plans, obey constraints, and recover from failure without turning the workflow into a procedural bonfire. ...

TL;DR for operators Pre-Act is a useful reminder that enterprise agents do not fail only because they choose the wrong tool. They fail because they lose the plot. A customer asks for help, the agent gathers one fact, calls one API, sees an unexpected result, and then behaves as if the workflow has reset. Charming, in the same way a lift that forgets floors is charming. ...