Stop Paying Twice for the Prompt: Preference Packing Reworks DPO’s Execution Layout

TL;DR for operators

A DPO-style preference pair usually contains one prompt and two ranked responses. Conventional execution turns that into two prompt-response sequences, which means the same prompt is processed twice.

Jaekyung Cho’s Preference Packing: Efficient Preference Optimization for Large Language Models¹ treats that duplication as a systems problem. It stores the prompt once, places the alternative responses behind it, and uses masking plus adjusted position IDs so each response still behaves as though it were paired independently with the prompt.

The reported payoff is material in the tested configurations. In a multi-node Qwen2.5-72B DPO run on Tulu3, preference packing alone reaches 1.53× normalized throughput relative to vanilla DPO. Batch sorting reaches 2.62×, while using both reaches 3.22×. The techniques are therefore complementary rather than substitutes.

The qualification is equally operational. Packing is not automatically cheaper. Dense-attention cost grows with the length of the combined sequence, so long responses can consume the savings created by removing duplicate prompts. Teams should evaluate the prompt-to-response length distribution of their own preference data before changing the training layout.

The duplicated work is inside the preference pair

Consider a preference example with one long instruction and two candidate answers. The learning objective needs to compare those answers, but conventional execution commonly represents them as two separate sequences:

prompt + preferred response

prompt + rejected response

The second sequence repeats the prompt even though nothing about that prompt has changed. When the input represents a large fraction of the token budget, this duplicates attention computation and associated memory by construction.

Preference packing changes the sequence layout. The prompt appears once, followed by all $K$ ranked responses.

Simply concatenating them would be incorrect, because later responses could then attend to earlier responses. The method therefore introduces a causal attention structure in which each response can see the shared prompt and its own preceding tokens, but not sibling responses. Position IDs are also adjusted so that a response retains the positional structure it would have had in its original prompt-response sequence.

Those two details are the mechanism. The method is not ordinary packing of unrelated examples to reduce padding. It exploits the specific structure of preference data: several responses share one input, yet their response-side computations must remain isolated.

Packing trades duplicated prompts for a longer sequence

Removing repeated input work sounds unconditionally attractive until attention scaling enters the calculation.

The paper models the dense-attention cost ratio as

Here, $l_{\mathrm{in}}$ is the prompt length and $l_{\mathrm{resp}}^{k}$ is the length of response $k$. Preference packing has strictly lower attention cost only when

This is the decision boundary that matters for adoption. Packing removes $K$ copies of the prompt, but it also constructs one longer attention sequence containing every response. Because dense attention scales quadratically with sequence length under the paper’s model, sufficiently long responses can outweigh the prompt savings.

That predicts a favorable regime: long shared inputs with relatively short responses.

The paper separately derives the attention-memory comparison under FlashAttention, where the modeled memory requirement scales linearly with sequence length:

Under the stated assumptions, sharing the prompt does not increase this attention-memory ratio and gives a strict advantage when response lengths differ.

Compute and memory therefore need separate evaluation. A workload can have an appealing memory profile without satisfying the same condition for dense-attention compute.

The experiments test resource efficiency, then scale

The paper’s three smaller-dataset experiments are the main evidence that the mechanism translates into runtime and memory differences. They use pairwise preference data with $K=2$ under single-GPU LoRA training.

The paper defines effective time to include the additional benefit available when lower peak memory permits a larger batch size. These values are best kept as the reported ratios: the source package notes that the paper’s separate statement about reducing memory “up to 80%” is not transparently reconciled with Table 1, so converting the table ratios into stronger percentage claims would overstate what the record supports.

The visual preference workload is particularly revealing. Its average input is about 600 tokens while the maximum response length is about 110. Image inputs expand into long token sequences, while answers remain comparatively short. That is close to the length profile the analytic argument predicts should benefit from sharing the prompt.

The large-scale experiment serves a different purpose: scalability and compatibility with another optimization. On Qwen2.5-72B-Instruct using more than 300,000 Tulu3 preference samples in a multi-node FSDP setup, throughput is normalized as follows:

Preference packing therefore does not need to displace length-aware batch sorting. In this experiment, the combined system is about 23% faster than batch sorting alone.

For training teams, profile tokens before rewriting the pipeline

What the paper directly shows: within the tested text, visual, single-GPU, and distributed configurations, eliminating repeated prompt processing can reduce resource use and improve throughput. It also shows analytically that the result depends on sequence geometry rather than on packing alone.

Cognaptus inference: infrastructure teams running substantial DPO or related preference-optimization workloads should treat shared-prompt duplication as a measurable cost category. Before implementation, compute prompt lengths, response lengths, their variance, and the fraction of each training sequence occupied by the input. A multimodal pipeline with expensive encoded inputs and short preference answers is a stronger candidate than a reasoning dataset dominated by long completions.

The method also suggests a layered optimization strategy. Batch sorting attacks inefficiency across examples by grouping similar sequence lengths. Preference packing attacks duplication within a preference example. The 72B result indicates that those optimizations can stack.

Lower memory pressure may have value beyond a lower memory statistic itself. For a team constrained by fixed accelerator capacity, it can create room for larger effective batches or improve utilization. The paper’s effective-time measurement is evidence for this pathway in its tested setups, not evidence that every cluster will realize the same economics.

Long reasoning responses are the main warning sign

The most consequential boundary is already contained in the cost equation. As responses become long relative to the prompt, the packed sequence grows, and its quadratic dense-attention cost can eliminate or reverse the advantage from sharing the input. The paper explicitly identifies reasoning models with extremely long responses as a difficult regime.

Three other limits constrain how far to generalize the result.

First, the public preference datasets in the main experiments contain pairs, so $K>2$ is analyzed theoretically but not empirically demonstrated. Second, the evaluation measures resource efficiency rather than downstream preference quality, win rate, or long-horizon convergence. The mechanism is intended to preserve response-wise computation, but this study does not provide a broad quality-equivalence study. Third, the hardware and execution evidence covers the reported single-GPU LoRA and multi-node FSDP setups, not every training stack.

The adoption question is therefore narrower than whether preference packing is “better.” For a training team, it is whether enough of the current GPU bill comes from repeatedly processing long shared inputs to justify changing the sequence layout.

When that condition holds, the paper identifies an unusually concrete source of waste: the model may be paying multiple times to read the same prompt before it learns anything from the preference between the answers.

Cognaptus: Automate the Present, Incubate the Future.