Time to Prefer: Why Binary RLHF Feedback Leaves Reward Models Guessing Thumbs-up feedback looks efficient. It is clean, cheap, easy to store, and friendly to dashboards. One output wins, another output loses, and the reward model learns what humans supposedly want. A tidy little morality market, with all the nuance of a vending machine. ...
Lunch is a simple word. In an AI assistant demo, “order me lunch” looks like the kind of request that should be easy by now. Open the food app. Pick something. Pay. Done. The button-clicking part is no longer the miracle. The problem is everything the user did not say. Do they avoid peanuts? Do they usually order from Tuantuan or Chilemei? Is “light lunch” about calories, price, time, or avoiding the food coma before a meeting? Should the assistant ask first, or does asking defeat the whole point of assistance? And if the user says no, does the assistant actually stop, or does it “helpfully” continue doing the wrong thing with the confidence of a junior consultant holding a fresh slide deck? ...
Memory sounds simple until a business actually needs it. A sales agent should remember what the client objected to last month. A customer-support agent should remember that a refund exception was already approved. A research assistant should remember which dataset was rejected, not vaguely summarize it into “user prefers cleaner data.” A healthcare or financial assistant should not turn a precise historical statement into a soft personality trait because the memory layer wanted to look elegant. Cute demos tolerate this. Production systems do not. ...
Files are where AI agent demos go to become adults. In a product video, the agent opens a few clean documents, remembers your preferences, drafts an answer, books the meeting, and looks quietly inevitable. In an actual computer, the same agent faces a folder called final_final_v3, a receipt saved as an image, a calendar invite with the wrong title, a video that contains the decisive evidence at second 8, and three people who all appear in the same user’s digital life. Suddenly the assistant that “knows you” looks less like a colleague and more like an intern who has discovered search for the first time. ...
Car settings are usually pretending to know you. Sport mode assumes you are impatient. Eco mode assumes you have discovered moral superiority through fuel efficiency. Comfort mode assumes everyone in the vehicle prefers to be gently transported like a bowl of soup. These modes are not useless. They are just blunt. They adjust a handful of parameters and call the result personalization, which is a bit like calling a restaurant “personalized” because it offers small, medium, and large. ...
The problem starts with a very ordinary sentence “Order my usual lunch.” For a human assistant, this sentence is not empty. It carries history. It points to an app, a restaurant, a branch, a meal, maybe a delivery address, maybe a payment method. For a conventional GUI agent, it is a trap wearing casual clothes. ...
Memory sounds simple until the assistant has to remember two incompatible things at once. A customer loves craft beer. The same customer is temporarily taking antibiotics. A flat memory system retrieves “likes IPA” and recommends a variety pack, because apparently “memory” means grabbing the loudest sticky note from a drawer and pretending it is wisdom. A more useful assistant retrieves the preference, the medical constraint, the timing, and the relation among them. It recommends a mocktail and quietly avoids turning personalization into negligence. ...
Chess engines are very good at telling you what a player should do. That is not the same as predicting what the player will do. Anyone who has watched a beginner hang a queen, an intermediate player force a dubious attack, or a strong player choose a quiet positional squeeze already knows the difference. Optimality is one question. Human behavior is another. Most AI systems enjoy pretending those two questions are basically cousins. They are not. One is about the board. The other is about the person touching the pieces. ...
Opening — Why this matters now As large language models evolve from word predictors into behavioral simulators, a strange frontier has opened: synthetic humanity. From virtual therapists to simulated societies, AI systems now populate digital worlds with “people” who never existed. Yet most of these synthetic personas are shallow — a few adjectives stitched into a paragraph. They are caricatures of humanity, not mirrors. ...
🧠 From Freud to Fine-Tuning: What is a Superego for AI? As AI agents gain the ability to plan, act, and adapt in open-ended environments, ensuring they behave in accordance with human expectations becomes an urgent challenge. Traditional approaches like Reinforcement Learning from Human Feedback (RLHF) or static safety filters offer partial solutions, but they falter in complex, multi-jurisdictional, or evolving ethical contexts. Enter the idea of a Superego layer—not a psychoanalytical metaphor, but a modular, programmable conscience that governs AI behavior. Proposed by Nell Watson et al., this approach frames moral reasoning and legal compliance not as traits baked into the LLM itself, but as a runtime overlay—a supervisory mechanism that monitors, evaluates, and modulates outputs according to a predefined value system. ...