The Robot Finished the Task, but Not Your Way

TL;DR for operators

A household robot can put an object in the correct place while ignoring instructions about how to handle it. A dashboard reporting only task completion would count that operation as successful, even when the user explicitly requested a different handling method.

LIBERO-PeRM makes that discrepancy measurable. In one zero-shot evaluation, a vision-language-action model completed 97.50% of motion-preference tasks, yet its preference-satisfaction score was only 24.38%. Skill-preference tasks showed a similar gap: 100.00% task success against 25.00% satisfaction.

For robotics product managers, personalized manipulation therefore needs separate acceptance criteria for completing the task and respecting the user’s instructions. Fine-tuning can substantially improve preference adherence, but training strategies trade off general-task retention, execution styles, and contextual language understanding. Broader demonstration coverage helps, while unfamiliar tasks, combined preferences, and conflicting instructions expose persistent weaknesses.

These findings come from controlled simulation, not physical household deployment. Their immediate value is a more discriminating evaluation framework.

Measuring what happens between instruction and completion

Li and colleagues introduce LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation¹ to evaluate whether robots respect explicit preferences while achieving an unchanged task goal.

Built on the LIBERO simulation environment, the benchmark contains 312 tasks and approximately 40.8K procedurally generated demonstration episodes. It covers 37 individual preference instances, 18 two-preference combinations, and five evaluation suites. Four suites account for the 312 tasks; the fifth, ComprePref, tests contextual-language formulations of existing tasks rather than adding new task combinations.

Preferences fall into three categories:

• Motion Preference: How the robot moves, including speed, acceleration, trajectory height, or proximity to other objects.
• Skill Preference: How the robot manipulates an object, including grasp direction, using a handle, pushing instead of lifting, or rotating during placement.
• User Profile: Which object or destination the robot selects, such as favoring a particular color or placing certain items in a specified drawer.

This classification matters because the required behavioral change occurs at different stages of execution. Motion constraints affect trajectories, skill preferences affect manipulation strategies, and profile preferences affect subtask planning and target selection.

The benchmark measures two outcomes. Task success rate (SR) records whether the original goal was achieved. Preference satisfaction (SA) records adherence to the requested behavior using four discrete scoring levels: 0, 0.25, 0.75, and 1.

Crucially, SA is gated by task success. An episode that fails the original task receives zero satisfaction, even if parts of its execution followed the preference. SA therefore measures successful, preference-compliant execution rather than preference compliance independently of task completion.

That distinction helps explain the zero-shot results reported in Section 4.2, Table 2.

The model’s no-preference task-success rate was 97.60%.

On motion and skill tasks, successful completion frequently occurred without satisfying the requested execution constraints. User Profile preferences presented a different problem: the model often failed the task itself, consistent with difficulty selecting appropriate objects or targets.

The six-model zero-shot comparison establishes the benchmark’s central diagnostic finding. Existing manipulation competence cannot be treated as evidence of personalized execution.

Preference training improves compliance but changes other capabilities

Once the failure is measurable, the next decision is how to train against it.

The researchers fine-tune pi_0.5 using low-rank adaptation (LoRA) and compare three strategies: learning from preference demonstrations alone, training general LIBERO tasks and preferences together, or learning general tasks before subsequently adapting to preferences.

All three start from the same base checkpoint.

Preference-only fine-tuning raises aggregate in-domain satisfaction to 82.48%, compared with 37.92% in the reported zero-shot setting. Explicit preferences can therefore be learned effectively within familiar simulated conditions.

But the training comparison reveals more than a simple improvement.

Source: Section 4.2, Table 3. Values are means across 10 independent evaluation runs using fixed model checkpoints. The reported variation does not measure independently repeated training.

Mixed training preserves substantially more general-task capability than sequential adaptation: 91.00% versus 52.33% no-preference task success.

It also improves satisfaction on contextual prompts, which convey known benchmark preferences through less direct language. Mixed training reaches 70.07% compared with 42.64% for preference-only training. This tests recognition of contextual expressions of predefined preferences, not inference of an individual’s previously unobserved preferences.

However, mixed training reduces Motion Preference satisfaction from 85.37% to 66.06%. The strategy that better retains general manipulation and handles contextual instructions performs worse on this specific execution category.

Cognaptus inference: A robotics engineering team selecting an adaptation strategy should first identify which capabilities must survive personalization. For a system where motion characteristics are consequential, aggregate satisfaction and general-task retention cannot substitute for motion-specific testing.

The reported comparison concerns these pi_0.5 training configurations. It does not establish that mixed training necessarily harms motion learning under other datasets, architectures, or training recipes.

More varied demonstrations improve learning, but transfer remains weak

Training performance also depends on what the demonstrations cover.

The benchmark’s data-scaling experiments compare configurations using a common 2.25K-demonstration budget while varying layout coverage and the inclusion of no-preference examples.

In Figure 4, distributing demonstrations across more layouts improves preference learning more than concentrating them within fewer layouts. Adding matched no-preference demonstrations produces further improvements without increasing layout diversity.

These are controlled data-composition comparisons, not a separate claim that collecting more demonstrations always solves personalization.

One possible explanation is that matched task examples help distinguish the original task requirement from the preference-dependent behavior. The experiments show the performance benefit; they do not establish that the learned policy internally represents these factors separately.

The generalization tests impose a stronger challenge. When the same task and preference appear in an unseen layout, the preference-only model records 28.36% satisfaction, compared with 82.48% in the reported in-domain evaluation (Table 3). Performance remains limited when both the task and layout differ from training conditions.

For robot-learning teams budgeting demonstration collection, broader layout coverage and matched preference/no-preference examples are promising choices when the target resembles these simulated manipulation tasks. Physical data-collection costs and real-world benefits remain unmeasured.

A model that imitates preferences within familiar scenes has not necessarily acquired transferable preference competence.

Combining preferences creates a new failure mode

Personalization becomes more demanding when users specify several requirements simultaneously.

LIBERO-PeRM separately tests whether a model can combine individually learned preferences and whether it can recover an individual preference from training involving combined conditions.

In the tested two-preference composition setting, evaluation reaches 25.0% task success and 25.0% satisfaction. In the reverse decomposition setting, task success reaches 37.5%, but satisfaction is 0.0% (Section 4.2, Table 4).

These experiments diagnose compositional generalization rather than isolate a training component through ablation. Their results indicate that learning individual or joint conditions does not reliably produce reusable preference components.

Conflicting instructions expose another limitation.

Under task-preference conflicts, the evaluation finds:

• 33.3% of trials follow the task while violating the preference.
• 29.2% follow the preference while violating the task.
• 37.5% complete neither.

In separate preference-preference conflicts, satisfaction falls from 63.40% in the reported in-domain reference condition to 47.75% under conflict. The benchmark counts satisfaction of either conflicting preference when the task succeeds.

These scoring conventions matter. The tests measure execution outcomes under contradictions; they do not establish whether a model recognizes the conflict, assigns priorities correctly, or knows when to request clarification.

For teams specifying persistent user settings, conflict resolution therefore requires its own behavioral requirements. A robot should not be assumed to resolve contradictory settings merely because it follows each setting successfully when presented alone.

A deployment evaluation must extend beyond the benchmark

LIBERO-PeRM provides moderate evidence for diagnosing preference-following failures within standardized simulation. It combines controlled task conditions, automated evaluators, multiple baseline policies, fine-tuning comparisons, and initial human review.

Its external-validity boundary is substantial.

All manipulation experiments use LIBERO-derived simulation. Demonstrations rely partly on generated plans, heuristic skills, and motion planning, while preference metrics depend on observable geometric, kinematic, visual, or subtask outcomes.

An initial survey of 15 participants rates demonstration human-likeness at 9.11/10 and preference plausibility at 8.35/10. Those ratings support initial content validation, not physical reliability or long-term user satisfaction.

The benchmark also excludes learning preferences from interaction history, changes in user preferences over time, and implicit or subjective preferences that cannot be expressed and automatically measured within its setup.

For a robotics product manager defining acceptance criteria, Cognaptus would translate the findings into four separate evaluation requirements: task completion; category-specific preference satisfaction; performance under unfamiliar tasks and layouts; and behavior when instructions conflict. Training-strategy comparisons should also verify retention of general-task capabilities.

These are proposed engineering requirements informed by the benchmark, not commercially validated thresholds. Physical safety, real-user interpretation, and conflict clarification need additional testing beyond LIBERO-PeRM.

Personalization must be demonstrated in the execution

The benchmark’s main contribution is making a previously obscured failure measurable. A robot can achieve the requested final state while disregarding the constraints that make its behavior acceptable to a particular user.

Preference fine-tuning can reduce that gap under familiar conditions, but the results also show losses in general-task retention, category-specific behavior, and transfer when the training strategy or environment changes.

For personalized robotics, the acceptance question is not simply whether the robot finished the task. It is whether it finished the task in the requested way, under the conditions the product is expected to encounter.

LIBERO-PeRM supplies a controlled foundation for asking that question. Reliable answers in physical environments still require independent evidence.

Cognaptus: Automate the Present, Incubate the Future.