Aligned or Just Agreeable? Why Accuracy Is a Terrible Proxy for AI–Human Alignment

Accuracy is comforting because it gives us a number. The model predicted the right label. The chatbot chose the same option as the survey respondent. The simulated customer picked the same product. Everyone claps, someone updates a dashboard, and the alignment problem is declared mostly solved.

Unfortunately, decision-making is where accuracy goes to look respectable while quietly doing very little.

A language model can produce a plausible schedule, a reasonable budget, or a believable recommendation while using a completely different internal weighting of the factors that shaped the human decision. It may arrive at the same surface answer for the wrong structural reason. Or worse, it may look “human-like” in aggregate while flattening the differences that matter most across subgroups.

That is the problem behind XChoice, a new framework for evaluating AI–human alignment in constrained choice decision-making.¹ The paper’s central move is simple but important: stop asking only whether the model chose the same outcome as humans, and start asking whether the model appears to use the same trade-off mechanism under the same constraints.

That shift sounds technical. It is. But it is also deeply practical. Most business uses of AI are not open-ended philosophical conversations. They are constrained choices: allocate time, rank options, assign budget, schedule workers, recommend products, prioritize leads, simulate customers, or decide which task deserves attention first. In these settings, the model’s answer is less important than the implied exchange rate behind the answer.

If your AI assistant says a worker should spend four hours on client calls, three hours on research, and one hour on admin, you do not just want the numbers. You want to know what the system implicitly values: client urgency, employee fatigue, revenue impact, contractual risk, fairness, or whatever else was quietly converted into a schedule. Alignment lives in that conversion, not in the final spreadsheet cell.

The alignment problem is not the answer; it is the exchange rate

XChoice begins from a familiar economic idea: people make decisions under constraints. A day has 1,440 minutes. A budget has a ceiling. A team has finite capacity. A shopper has preferences, prices, and a wallet. A manager has deadlines, staffing limits, and political headaches, which sadly are not yet deprecated.

In a constrained choice problem, a person does not simply “choose” an outcome. They allocate scarce resources across competing options. That allocation reflects a set of trade-offs. More sleep usually means less work or less leisure. More budget for one product category means less budget somewhere else. More attention to one client means less attention to another.

Outcome-based alignment metrics usually ignore this structure. They compare answers. XChoice compares estimated decision mechanisms.

The paper formalizes the decision-maker as solving a constrained optimization problem. In the general framework, a person or model chooses a vector $x$ to maximize an objective function subject to equality and inequality constraints. The parameters of that objective function represent the trade-off weights: how much the decision-maker appears to value different factors when constraints force substitution.

In the paper’s case study, the constrained resource is time. The authors use the American Time Use Survey as the human baseline and ask several LLMs to allocate a typical day across four categories:

The final analytic sample contains 4,307 people after preprocessing. Each individual has demographic and socioeconomic attributes such as age, education, weekly earnings, sex, spouse or partner status, and race indicators. The models receive profile-based prompts and generate time allocations in minutes across the same four categories.

The obvious evaluation would compare generated time allocations with human time allocations. XChoice instead estimates, for humans and for each model, the parameters that make those allocations look optimal under the time-budget structure.

That is the key. The paper does not treat the LLM as a mystical black box with a daily planner habit. It treats the LLM output as observed behavior and asks: what preference weights would have produced this behavior if it came from a constrained decision-maker?

The mechanism is recovered before alignment is judged

The method works in three steps.

First, define the constrained decision problem. In the time-use case, each person distributes a fixed daily budget across leisure, work, sleep/personal care, and other activities. The model uses a utility function in which demographic attributes influence the relative value of each activity. “Other” is used as the reference category, so coefficients for leisure, work, and sleep/personal care are interpreted relative to that baseline.

Second, estimate the parameters. The authors use a structural estimation approach: fit the model-predicted time shares to observed time-use shares, using nonlinear least squares. This is done for the human ATUS data and separately for the time allocations generated by each LLM.

Third, compare the recovered parameter vectors. XChoice uses cosine similarity to compare the direction of model and human parameter vectors, and mean absolute deviation to measure coefficient gaps. At the attribute level, it aggregates deviations to identify which features systematically drive misalignment.

That gives a more useful object than “the model was 73% accurate,” because the output is interpretable:

This is the business-relevant distinction. A model can be outcome-aligned in a narrow benchmark while being mechanism-misaligned in a way that fails under new constraints. It may recommend the same product today but for a different reason. Change the budget, the customer segment, or the policy environment, and the hidden difference matters.

This is also why the paper’s use of the Lucas critique is not academic decoration. Reduced-form correlations are fragile when the environment changes. Structural parameters are supposed to be more stable because they describe the underlying trade-offs rather than the surface relationship observed in one dataset. “Supposed to be” is doing work here, of course. Structural models can be misspecified. But the premise is sensible: if we care about generalization under changed constraints, mechanism-level alignment is more informative than answer-level mimicry.

The model rankings are less interesting than the pattern of failure

The paper evaluates five LLMs: GPT-4o, Claude-3.7, DeepSeek-V3, Llama-3.3, and Qwen-2.5. The model-level results are already useful, but they are not the main intellectual prize.

Claude-3.7 is the strongest overall in the reported alignment diagnostics. It has the smallest mean absolute deviation from the human baseline, at 0.094, and the highest work-alignment cosine similarity, 0.78. GPT-4o is moderately positive across activities, while DeepSeek-V3 has positive alignment for leisure but negative alignment for sleep/personal care and work. Llama-3.3 and Qwen-2.5 show larger departures, with Qwen-2.5 negative across all three focal activities in the cosine-similarity heatmap.

A lazy reading would turn this into a leaderboard. Claude wins, Qwen loses, please subscribe for more AI benchmark theater.

That would miss the point.

The paper’s important result is not only that some models are “more aligned” than others. It is that alignment is activity-dependent and attribute-dependent. A model can track human-like weighting in one activity and miss it in another. It can be tolerable at the aggregate level and still encode very different subgroup trade-offs.

This matters because many practical AI deployments are segment-sensitive. A customer simulation tool that behaves plausibly on average but compresses subgroup differences can mislead pricing, marketing, public service planning, or workforce policy. The danger is not always an obviously absurd output. The danger is a smooth average that erases the very heterogeneity the system was supposed to help analyze.

The subgroup results show why plausible schedules are not enough

The paper’s most concrete misalignment diagnostics concentrate around two attributes: Race: Black and Spouse: Present. The authors also report high deviation for Pacific Islander and Native American indicators, but they explicitly treat those categories cautiously because sample sizes are small and uncertainty is large. Good. Not every large coefficient gap deserves a dramatic subplot.

For Race: Black, the human coefficients are positive and statistically significant across leisure, sleep/personal care, and work. Under the paper’s normalization, this means Black respondents in the ATUS sample allocate more time to each focal activity relative to the reference category “Other,” compared with the White reference group. The reported human coefficients are:

Most LLMs attenuate or reverse these effects. In leisure, for example, DeepSeek-V3 is -0.119, Claude-3.7 is -0.100, and GPT-4o is -0.053. These are not tiny stylistic differences in how the model formats a schedule. They imply different trade-off weights.

For Spouse: Present, the human coefficients are negative and statistically significant across the same activities:

The interpretation is again relative to “Other.” In the human data, people with a spouse present shift time away from the focal categories and toward the reference category. Several models shrink these negative effects sharply toward zero. GPT-4o, for example, is close to zero for work and sleep/personal care, producing large deviations in those rows.

Here is the deeper point: the models are not merely “wrong” in a generic sense. They appear less sensitive to the demographic attributes that shape human time allocation. The appendix’s distributional plots reinforce this. Human time shares are diffuse, heavy-tailed, and heterogeneous. LLM-generated shares are more concentrated, often heaped around a few focal values, and they under-allocate the residual “Other” category. In plain English: the models often produce template days.

Template days are seductive because they look tidy. Humans are not tidy. Anyone who has seen an actual calendar knows this, but apparently our evaluation metrics needed a reminder.

The appendix distribution plots are not decoration; they explain the mechanism gap

The appendix adds a distributional view across race and spouse/partner status. Its likely purpose is not to introduce a second thesis. It supports the main mechanism argument by showing what the coefficient diagnostics are picking up.

The human distributions have wider support and heavier tails. The LLM distributions cluster around focal values. This is consistent with round-hour reasoning: eight hours of sleep, eight hours of work, a little leisure, a token “other.” That kind of output is plausible enough to pass a casual sanity check. It is also too compressed to represent the messiness of real time-use data.

This matters because time allocation is full of residual categories. Commuting, caregiving, errands, household tasks, waiting, fragmented obligations, and small interruptions do not always fit neatly into “work,” “leisure,” or “sleep.” In the paper’s results, LLMs tend to under-allocate “Other,” which means they may over-commit to salient categories while missing the everyday friction that fills human lives.

For business applications, this is a warning about AI simulations. If a model simulates customers, employees, patients, or citizens with too-clean schedules, it may understate operational friction. It may overestimate available attention, overpredict willingness to engage, or assume that people can reallocate time more easily than they actually can.

That is not a small modeling error. That is the difference between designing for humans and designing for a spreadsheet’s imaginary roommate.

Reduced-form checks are useful, but they do not carry the constraint

The paper compares XChoice estimates with reduced-form OLS regressions. This test has a clear purpose: it asks whether the structural mechanism is just reproducing ordinary regression coefficients with extra ceremony.

It is not.

The coefficient scatterplot shows substantial dispersion between OLS and XChoice estimates. Many coefficients that appear large under OLS shrink toward zero under the structural model. The authors interpret this as evidence that reduced-form regressions can amplify activity-specific correlations, while XChoice produces more conservative weights that account for substitution under the fixed time budget.

That distinction is important. In a time allocation problem, activities are not independent. More of one activity mechanically means less of something else. A regression on one activity’s time share can capture association, but it does not directly encode the adding-up constraint across activities. XChoice builds that constraint into the model.

The robustness test then applies mild covariate shifts: an earnings lift, an age-band shift, a race-mix shift, and a spouse-mix shift. The purpose is robustness/sensitivity testing, not main evidence. The question is whether the estimated parameters drift less under small environmental changes than reduced-form OLS coefficients.

They do.

This does not prove that XChoice is universally invariant. The shifts are mild, the structural form is assumed, and large institutional changes could still move the parameters. But the test supports the paper’s narrower claim: in this setting, the structural estimates behave more like stable trade-off weights than the reduced-form coefficients do.

That is exactly the kind of evidence a business evaluator should want. Not “is this model accurate in yesterday’s test set?” but “does the diagnostic still make sense when the environment changes a little?”

The RAG intervention is useful because it sometimes fails

The paper also uses XChoice as a guide for targeted mitigation. This is where the framework becomes more than a diagnostic report. Once XChoice identifies the attributes where model-human trade-offs diverge, the authors try a retrieval-augmented generation intervention.

The RAG setup is focused on the two most consistently divergent attributes: Race: Black and Spouse: Present. It targets GPT-4o and DeepSeek-V3 as representative closed- and open-source models. The knowledge base contains structured instances from empirical studies: 50 instances on marital status and time allocation, and 43 on race and time use. The system embeds respondent personas and retrieves the top three relevant knowledge instances for each case, then re-estimates alignment after the model generates new time allocations.

The results are mixed, which makes them more useful.

For GPT-4o, RAG improves both targeted subgroup alignments, especially Spouse: Present. For DeepSeek-V3, RAG improves Race: Black but damages Spouse: Present, where baseline alignment was already relatively strong.

This is the most business-relevant result in the paper.

RAG is often sold as a universal cure for hallucination, domain knowledge gaps, and general organizational anxiety. Here, it behaves like an intervention with side effects. It helps when the model has a real mechanism gap that retrieved evidence can correct. It can hurt when the model was already close and the new evidence perturbs an existing trade-off pattern.

That suggests a practical rule: diagnose first, inject second.

Do not automatically stuff every decision prompt with domain knowledge because the architecture diagram looks more professional with a retrieval box. Use mechanism-level diagnostics to identify where the model’s implied trade-offs diverge, then target retrieval to those gaps. And after intervention, re-estimate the mechanism instead of celebrating because the prompt got longer.

Long prompts are not governance. They are sometimes just expensive confetti.

What XChoice directly shows, and what business should infer carefully

The paper directly shows three things in its chosen setting.

First, LLMs differ meaningfully in mechanism-level alignment with human time allocation. Claude-3.7 is strongest among the evaluated models by the reported diagnostics, while Qwen-2.5 and Llama-3.3 show larger departures.

Second, misalignment is not uniform. It varies by activity and attribute. Race: Black and Spouse: Present are especially salient in the main analysis, while small-sample race indicators require caution.

Third, targeted RAG can improve alignment where a mechanism gap is visible, but can also degrade alignment where the model was already close.

The business inference is broader but should remain disciplined. XChoice suggests an audit pattern for AI systems that support or simulate constrained decisions:

The inference is not that every company should copy the exact time-use utility function from this paper. Please do not make your CRM allocate customers’ emotional lives into “work, leisure, sleep, and other.” The lesson is architectural: when an AI system makes constrained choices, the evaluation layer should recover and compare the implied trade-offs, not merely score the visible answer.

The operational value is diagnosis, not a prettier benchmark

For organizations, XChoice is most valuable as a governance pattern.

A practical implementation would look like this:

1. Define the constrained decision class.
2. Collect or identify reliable human baseline decisions.
3. Specify an interpretable structural model of the decision.
4. Estimate human trade-off parameters.
5. Estimate model trade-off parameters.
6. Compare mechanisms by model, option, and subgroup.
7. Apply targeted interventions only where gaps are visible.
8. Re-estimate after intervention.

That workflow is slower than asking an LLM judge whether the output “looks reasonable.” It is also less silly.

The gain is not just better academic measurement. It is cheaper failure localization. If a model underperforms, XChoice can help identify whether the issue is a model-level weakness, an activity-specific mismatch, a subgroup-specific distortion, or an intervention side effect. That matters for ROI because remediation becomes more precise.

A company does not need to know that “alignment is low” in the abstract. It needs to know whether the model undervalues residual time, compresses segment variation, ignores marital-status trade-offs, overweights income, or collapses heterogeneous customers into one tidy template. Those are fixable problems. “The model is not aligned” is a mood.

Boundaries: structural alignment is not magic, and subgroup coefficients are not destiny

The paper’s limitations are important because they define where this framework should not be oversold.

XChoice requires decisions that can be represented as constrained optimization problems with an explicit objective, feasible set, and interpretable trade-offs. That covers many business decisions, but not all AI behavior. Open-ended writing, negotiation, creative reasoning, and multi-turn assistance may not fit cleanly into this structure without forcing the problem into a costume it did not ask to wear.

The approach also inherits the risks of structural estimation. The recovered parameters depend on the assumed functional form, the normalization choice, the observed covariates, and the available human baseline. If the model is misspecified or important variables are omitted, the estimated weights can mislead. The framework is interpretable, not omniscient.

Subgroup analysis needs particular care. The coefficients are descriptive estimates in a dataset and model, not normative claims about how any group “should” behave. The paper is appropriately cautious about categories with small sample sizes and wide uncertainty. Business users should be even more cautious, because operational decisions based on subgroup coefficients can easily drift into stereotyping, discrimination, or manipulative personalization.

There is also a dual-use risk. A mechanism-level alignment tool can help audit models for fairness and reliability. It can also help tune systems to imitate human decision patterns more effectively for persuasion or manipulation. That is not a reason to avoid diagnostics. It is a reason to govern who uses them, for what purpose, and with what human review.

Accuracy is a mirror; mechanism is the machinery

The best way to read XChoice is not as another LLM benchmark. It is a reminder that alignment in decision-making is not the same as agreement.

Agreement is a mirror. It reflects whether the output resembles the human answer in a particular setting.

Mechanism is the machinery. It tells us how the system turns attributes, constraints, and trade-offs into a decision.

For simple classification tasks, the mirror may be enough. For constrained decisions that affect resource allocation, customer treatment, scheduling, or behavioral simulation, the machinery matters more. A model that gives a plausible answer while using the wrong exchange rates is not aligned. It is just agreeable.

And agreeable systems are dangerous precisely because they do not look broken.

They look helpful. They look reasonable. They produce clean schedules, neat budgets, and confident recommendations. Then the environment shifts, the subgroup changes, the constraint tightens, or the intervention adds retrieval noise, and the hidden trade-off pattern finally becomes visible.

XChoice gives us a way to inspect that pattern before deployment, not after the dashboard starts lying politely.

Cognaptus: Automate the Present, Incubate the Future.