The Cost of Asking Only for Today

TL;DR for operators

A persistent assistant can miss valuable information if it asks about a preference only when the answer improves its current task enough to justify the interruption. Wang and colleagues investigate this problem through Active Preference Acquisition (APA), which values questions according to expected benefits across future tasks.¹ In their synthetic benchmark, 34.4% of budget-feasible valuable initially missing preferences at a 20-task horizon are worth acquiring only through cumulative reuse. APA acquires 80.3% of these preferences versus 5.1% for current-task valuation, with a statistically significant net-utility advantage among affected users. However, its overall 10-task advantage is statistically inconclusive after adjustment, and a privileged oracle achieves better utility while collecting fewer preferences. The operational lesson is to select questions by expected downstream value, not to maximize the number of answers collected.

A preference can be too small to ask about today

Consider an assistant that repeatedly books travel, recommends restaurants, and schedules meetings. A preference about seating, spending, or meeting arrangements might improve each individual decision only slightly. Asking about it during any particular request could cost more in user effort than the immediate benefit justifies.

But the answer might influence ten subsequent decisions. The interruption happens once; the benefit can recur.

In Wang et al.’s ActiveWebPref benchmark, researchers identify 198 such user-attribute pairs among 576 budget-feasible valuable initially missing pairs at a 20-task horizon. That is 34.4% of the relevant set—not 34.4% of all preferences or all users.

The researchers call these amortization-dependent preferences: under a fixed initial profile, no individual future task provides enough expected benefit to cover acquisition cost, yet the accumulated benefit across tasks does.

This creates a specific failure mode for current-task Value of Information (VoI). If every encountered marginal benefit remains below the cost of asking, a policy concerned only with the current decision will never acquire the preference, even when repeated future use would repay the cost.

The fixed-profile qualification matters. Once an agent learns other attributes, a preference’s marginal value can change. The benchmark’s initial classification therefore does not guarantee that current-task VoI will never acquire a designated preference.

Prospective value changes what a question is worth

The paper formalizes Prospective Personalization Value (PPV) as the expected improvement across future, not-yet-arrived tasks, minus the weighted costs of asking and disclosing information.

Those costs are separate. Asking consumes user effort even when a request is refused; successfully acquiring sensitive information creates an additional modeled disclosure cost.

The relevant valuation is therefore:

Expected cumulative future task benefit − weighted asking cost − weighted disclosure-risk cost.

This extends conventional information-value reasoning from an immediate decision to a persistent user-agent relationship.

To approximate that quantity without knowing the user’s future tasks or hidden preferences, APA combines four operations.

First, an LLM identifies potentially missing preference attributes. Second, another component estimates the distribution of future task families. Third, the system evaluates counterfactual task outcomes using public task prototypes and a shared solver. Finally, an adaptive greedy policy selects questions according to estimated marginal value relative to asking cost, within an interaction budget.

The prospective benefit estimator combines three quantities: expected occurrences of each task family, the fraction of prototypes where a preference matters structurally, and the estimated utility impact when it matters.

Unasked attributes remain eligible for reconsideration. Successfully acquired preferences enter the persistent profile, while known refusals are excluded from future proposals.

The framework also recognizes that acquiring several preferences jointly may produce overlapping benefits. Its greedy selection rule recalculates marginal value after answers arrive, although the paper does not establish an approximation guarantee for that algorithm.

The benchmark measures what the agent could have known

ActiveWebPref provides the controlled environment necessary to test these decisions.

It contains 180 original test users, each with 27 preference attributes and a saved sequence of 20 future tasks. The environment spans travel, dining, shopping, scheduling, and work activities, organized into 16 task families. Three public prototypes per family support counterfactual valuation.

The main evaluation covers the first ten tasks. Hidden synthetic profiles determine exact task utilities, and simulated users provide deterministic answers or refusals.

All practical policies share the task solver and underlying benchmark environment. The comparison includes current-task VoI using the same public belief and solver as APA, reducing differences attributable to those components.

The result is a controlled acquisition-policy comparison, not evidence from observed customer interactions or live websites.

Better acquisition coverage produces a conditional utility advantage

At 20 tasks, APA acquires 159 of the 198 designated amortization-dependent preferences, compared with just 10 for current-task VoI: 80.3% versus 5.1%.

More importantly, among the 120 original-test users possessing at least one designated preference, APA achieves 0.263 higher whole-sequence net utility. The paired 95% confidence interval is [0.071, 0.479], with Holm-adjusted p = 0.021.

That is evidence of a downstream benefit, not simply a difference in the number of questions asked.

The broader result is less decisive.

Source: Section 6.1, Table 2. Results cover 180 synthetic users. Net utility deducts weighted acquisition and disclosure costs from cumulative task utility. The oracle has information unavailable to practical policies; Ask-Everything is exempt from the standard acquisition budget.

Across all 180 users at ten tasks, APA improves mean net utility by 0.091 over current-task VoI. Its reported 95% confidence interval is [0.019, 0.174], but the Holm-adjusted p value is 0.0615. The difference does not meet the adjusted 0.05 significance threshold.

The contrast is informative: prospective acquisition demonstrates a stronger measurable benefit for users with amortization-dependent preferences than for the benchmark population as a whole.

Asking more can reduce the value of personalization

Ask-Everything achieves higher raw cumulative task utility than APA, but its acquisition costs push net utility below even Ask-Nothing.

APA recovers 84.2% of the raw task-utility improvement from exhaustive questioning while using only 27.6% of its questions.

Cost-matched comparisons strengthen the evidence for selective acquisition. Under per-user expenditure caps derived from APA, the framework exceeds four heuristic policies in net utility, including an advantage of 0.072 over Impact-only selection (Holm-adjusted p = 0.0025). These comparisons support APA’s combined selection approach; they do not isolate reuse-frequency weighting as the sole explanation.

The privileged oracle raises a further complication. At 20 tasks, it acquires only 44.9% of designated amortization-dependent preferences, compared with APA’s 80.3%, yet obtains greater attributed task-utility gain: 0.369 versus 0.307.

The oracle often waits until the first relevant task to ask. Acquiring information earlier offers no extra benefit when the questioning budget is not binding.

Moreover, APA receives a pre-sequence questioning opportunity that current-task policies do not. Their comparison therefore does not separately establish the benefit of earlier timing.

Acquisition coverage is a diagnostic of policy behavior, not the objective to optimize.

The ablations favor cost discipline over elaborate forecasting

The component tests reveal where the evidence is strongest.

Removing asking-cost penalties increases question volume to 1.79 times APA’s level and reduces net utility by 0.257 (Holm-adjusted p = 0.0075). This directly supports cost-aware questioning under the benchmark’s utility pricing.

Other additions have less demonstrated value. Replacing learned task-family probabilities with uniform weights changes net utility from 6.755 to 6.760, without a significant difference. Supplying the complete eligible preference catalogue likewise produces no measurable improvement.

APA’s estimated preference-value ranking also shows limited agreement with realized oracle values: Spearman correlation is 0.215, and NDCG is 0.681.

These findings suggest that correctly estimating which preferences deserve acquisition remains difficult. A more elaborate future-task forecast is not yet shown to solve that problem.

Removing the disclosure-risk penalty raises successful disclosure-risk cost from 0.058 to 0.092, although the associated net-utility difference does not survive Holm adjustment. This risk term remains an evaluation penalty, not a privacy guarantee.

What a persistent assistant should do differently

For product teams building recurring travel, shopping, scheduling, or customer-service agents, the paper suggests a concrete design hypothesis: maintain an inventory of missing, potentially reusable preferences and decide whether to ask, defer, or omit each one according to expected future benefit and acquisition cost.

The evaluation should follow acquired information into subsequent tasks. Relevant measures include realized task improvement, interruption cost, refusals, unnecessary queries, disclosure sensitivity, and net utility over repeated interactions. Profile completeness alone cannot establish that collecting information was beneficial.

Implementation should also distinguish preference acquisition from consent management. Sensitive-attribute restrictions, transparent profile records, deletion capabilities, and access controls require explicit product mechanisms beyond the numerical risk penalty studied here.

The evidence does not establish the approach’s effectiveness with real users. ActiveWebPref uses synthetic alternatives, deterministic preferences, and simulated responses. It does not adequately represent evolving preferences, ambiguous answers, repeated-question annoyance, or the complexities of live websites.

A production test would need to examine these behaviors before treating benchmark utility improvements as demonstrated customer benefit.

The better question is the one with verified future value

APA establishes a credible reason for persistent agents to look beyond the immediate task when valuing missing preferences. Its strongest evidence comes from preferences whose benefits accumulate through repeated use, where prospective acquisition changes behavior and improves net utility for affected synthetic users.

But its oracle comparison, modest overall result, and valuation diagnostics constrain the conclusion. Knowing more about a user is not automatically better, and collecting information sooner is not automatically optimal.

The practical research direction is more selective acquisition, better estimates of downstream value, and evidence that the answers collected actually improve future decisions after their costs are counted.

Cognaptus: Automate the Present, Incubate the Future.