When Helpful Starts Deciding for You

TL;DR for operators

For subjective personal questions, the main risk is not limited to whether an AI gives a wrong answer. It is also whether the system helps a user form a judgment or supplies the judgment for them.

Cheng and colleagues call the latter pattern LLM-as-oracle use.¹ Their evidence spans public conversation logs, donated longitudinal histories, a preregistered experiment, a randomized three-week interaction study, and response-policy interventions.

Three findings matter operationally. First, users delegate interpretation and judgment frequently and often underestimate how much they do so. Second, reliance is partly shaped by the system: perceived AI identity, warmer perceptions of AI, and experimentally varied model behavior all change oracle-style prompting. Third, models can be pushed toward supporting self-deliberation through structured prompting or DPO, but conventional reward models and immediate user preference can penalize those less-oracular responses.

For AI products handling personal interpretation, advice, coaching, education, or employee support, preservation of user deliberation should therefore be evaluated as a separate objective. The paper establishes effects on interaction style and model responses; it does not establish that oracle use causes long-run loss of autonomy, competence, relationships, or well-being.

When the assistant supplies the judgment

A user asks an assistant what a colleague’s message “really means.” Another asks whether ending a relationship would be justified. A third asks what career move to make next.

In each case, an assistant can respond in at least two fundamentally different ways. It can widen the user’s view by presenting interpretations, considerations, options, and consequences. Or it can decide what the situation means, which judgment is correct, or which action the user should take.

The distinction matters because subjective questions rarely have the kind of ground truth that makes conventional AI-reliance research straightforward. An answer can be articulate, internally coherent, and welcomed by the user while still shifting interpretive authority from the user to the model.

LLMs as Oracles: Reliance on LLMs for Subjective Personal Questions formalizes that allocation of authority. The framework is not a classifier for “bad advice.” It measures how work is divided between human and model.

The paper deliberately excludes activities such as factual questions, translation, creative writing, journaling, pure context sharing, and agentic tasks. Oracle use is narrower than “personal AI use” and does not imply that the model’s answer is factually false.

Users delegate more than they report

The authors apply validated LLM judges to several datasets. In both WildChat and ThoughtTrace, more than 90% of classified interpretation and normative-judgment prompts were oracle-framed. Action-oriented tasks were more mixed.

The longitudinal evidence also points upward. In donated ChatGPT and Claude histories from 52 users, covering 140,834 messages, the overall oracle proportion rose strongly over time. Interpreter and Decision-maker roles increased, while the non-oracle Strategist role declined. Public WildChat data showed similar increases for Interpreter and Evaluator roles.

Those trends are descriptive. Changes in available models and changes in who was using them cannot be cleanly separated from changes within individual users.

The more revealing result concerns self-perception. Only 12 of the 52 data-donation participants correctly identified their most frequent role. Observed oracle roles were frequently ranked low or omitted in self-reports.

That mismatch creates a product problem that ordinary satisfaction metrics will not detect. A user can experience an assistant as supportive while gradually delegating more judgment than they recognize.

The system changes how much authority users give it

Reliance is not simply a fixed user trait.

In a preregistered experiment with 520 participants, 20.0% of messages sent to a disclosed AI were oracle-style, compared with 11.1% when participants believed the interlocutor was human. At least one oracle message was sent by 65.4% of participants in the AI condition versus 51.2% in the perceived-human condition.

This experiment should not be read as a clean estimate of the effect of the “AI” label alone: both conditions ultimately involved LLMs, and response styles could differ. But it shows that the perceived identity of the interlocutor changes how users allocate judgment.

Within the AI condition, warmer and more positive perceptions were also associated with more oracle use. A one-standard-deviation increase in perceived warmth corresponded to 55% higher odds of sending an oracle message; the equivalent increase in valence corresponded to 34% higher odds. These are associations, not randomized effects.

The stronger causal result comes from a separate randomized three-week study of 1,015 participants. When the authors reanalyzed interactions by model behavior, oracle-use rates were 5.4% under sycophantic behavior and 5.5% under challenging behavior, compared with 4.4% under balanced behavior. Sycophantic interactions also showed a significant increase in oracle use over time.

The result is more specific than “agreeable models create dependence.” Both sycophantic and challenging behavior exceeded the balanced condition. Response policy itself can alter how users subsequently approach the system.

Preserving deliberation can score worse

If oracle behavior is partly produced by the model, it becomes an alignment target rather than only a user-education problem.

The authors test structured prompting that frames the user’s goal as self-discovery, fast answers, or learning. Self-discovery prompting consistently reduces oracular responses across the tested models. Fast-answer prompting pushes in the opposite direction. They also apply Direct Preference Optimization to four open-weight models using preferences for regenerated, less-oracular responses; the resulting reductions are comparable to structured prompting.

Then the evaluation conflict appears.

Larger reductions in oracularity often come with lower conventional reward-model scores and more clarifying questions. In a 200-person preference study, participants preferred baseline oracular responses when asked which answer they liked immediately. When asked which response better supported developing their own interpretation, they preferred the less-oracular DPO responses.

For product teams, those are different objectives. A response optimized for immediate decisiveness can outperform a response designed to preserve deliberation under one evaluation criterion and underperform under another.

Cognaptus infers a practical design rule: products handling subjective, high-delegation interactions should not collapse these objectives into a single “helpfulness” score. Teams can detect interpretation, normative judgment, or consequential decision prompts; route them toward contextualization, deliberation, or strategy; and separately measure whether the model preserves user participation in the judgment.

That does not mean every personal question should trigger refusal or prolonged questioning. The paper’s own results suggest the relevant variable is response policy: whether the model takes over the judgment or helps structure it.

What remains unproven

The paper measures prompting behavior and model responses, not the full downstream consequences of reliance. An oracle-style prompt does not show that a user accepted the answer, changed a belief, followed a recommendation, or experienced diminished autonomy.

Generalization also has limits. The participant studies are U.S.-based; the donated-history sample contains only 52 relatively heavy conversational-AI users; public datasets have their own selection biases; and the intervention experiments are evaluated mainly at the response level rather than through long-run behavioral outcomes.

So the operational claim should remain narrow. Product teams now have evidence that subjective judgment delegation can be measured, that users may misperceive it, that model behavior can influence it, and that response policies can reduce it. They do not yet have causal evidence that reducing oracle use improves long-term well-being.

That is enough to change evaluation practice. If an assistant is used where interpretation and judgment matter, “Did the user like the answer?” and “Did the model preserve the user’s role in forming the answer?” are not the same question.

Cognaptus: Automate the Present, Incubate the Future.