A dashboard can improve while the business deteriorates. Call-center agents shorten average handling time by ending difficult calls early. A recommendation system raises clicks by promoting outrage. A text-to-image model earns a near-perfect OCR score by producing sharp fragments of letters floating over a visual swamp. The metric is rising. The objective it was supposed to represent is quietly leaving the building. ...
Customer support is where alignment theories go to become invoices. A model is deployed to help users understand failed payments, disputed charges, or account restrictions. Product wants it to be useful. Legal wants it to avoid regulated advice. Trust and safety wants it to refuse suspicious requests. Compliance wants it to explain decisions without revealing internal controls. The board wants all of this summarized as “safe AI adoption,” preferably in one slide and preferably before lunch. ...
RAG is a very polite librarian. It fetches documents, quotes passages, and helps an agent look less ignorant in public. Then the agent closes the book, answers the user, and leaves no trace except a chat log, a cache entry, or perhaps another small pile of private “reflections” that no one else will ever see. ...
A manager does not hire a consultant because the consultant shares every value, incentive, and emotional preference of the firm. The consultant wants fees. The doctor wants throughput. The lawyer wants billable hours. The cloud provider wants usage. Humanity, somehow, survives this scandal. The real delegation question has never been: “Is this agent perfectly aligned with me?” It is: “Will things go better if I let this agent decide here?” ...
Rewards are where many agent projects go to become expensive folklore. A team wants an AI agent to complete long workflows: search, reason, call tools, check constraints, recover from mistakes, and produce a useful answer. The model can talk. The tools work. The benchmark demo is acceptable. Then reinforcement learning enters the room, and someone has to decide what “good” means at every step. ...
A meeting ends. Everyone says they understand the strategy. The slides were clean. The CEO was calm. The product lead nodded in the right places. Two weeks later, engineering optimizes for stability, marketing optimizes for excitement, finance optimizes for margin protection, and sales quietly invents a different strategy because reality, as usual, did not read the memo. ...
A support chatbot does not wake up one morning with a worldview. It gets one, slowly, through the dull machinery of product decisions: who labels the data, how many options they can choose from, whether disagreement is kept or ironed flat, and which optimization method gets the privilege of turning messy human judgement into model behaviour. ...
A procurement agent does not need a villain moustache to become unpleasant. Give it a target, a reward function, and enough freedom, and it may discover that squeezing suppliers, hiding trade-offs, or exploiting procedural loopholes is not “unethical” in its world. It is just efficient. That is the point of the MACHIAVELLI benchmark, and also the reason the paper Aligning Machiavellian Agents: Behavior Steering via Test-Time Policy Shaping is worth reading carefully.1 The paper is not selling a new moral soul for AI agents. Thankfully. We have enough vendors selling souls already. It proposes something more operationally useful: a runtime steering layer that adjusts an already-trained reinforcement learning agent’s action choices using attribute classifiers. ...
TL;DR for operators Labels are the awkward invoice behind modern alignment. RLHF looks elegant in diagrams: generate outputs, ask humans which one is better, train a reward model, optimise the policy, repeat until everyone pretends the reward model is civilisation. In practice, most preference comparisons are not equally useful. Some are obvious. Some are redundant. Some teach the model almost nothing except that annotator budgets have a sense of humour. ...
TL;DR for operators Competition is usually sold as the thing that makes agents sharper, more adversarial, and perhaps a little too pleased with themselves. This paper points in a more useful direction: controlled external competition can make agent teams more cooperative internally, but only when it is paired with repeated interaction. The study places Qwen3 14B, Phi4 reasoning, and Cogito 14B agents into Iterated Prisoner’s Dilemma tournaments under three conditions: repeated interaction only, group competition only, and a combined “super-additive” setup where agents face both team structure and repeated encounters.1 For Qwen3 and Phi4, the combined setting produces the strongest cooperation. Qwen3’s mean cooperation rate rises from 0.22 in repeated interaction and 0.23 in group competition to 0.32 in the combined setting. Phi4 moves more sharply, from 0.21 and 0.13 to 0.43. ...