Train Wide, Deploy Narrow: LoRA Rank Does Not Have to Be One Decision

TL;DR for operators

A production team may want every LoRA adapter to fit a small, uniform serving footprint. The usual response is to choose that small rank before training and optimize inside the resulting constraint. This paper shows that the training capacity and the deployment capacity do not always need to be identical.

LoRA-Squeeze, proposed by Vulić et al.,¹ trains or partially trains a standard LoRA at a higher rank and then reduces it to a smaller rank using an SVD-based transformation. On Gemma 3 4B text tasks, compressing rank 16 to rank 4 averaged 90.41% accuracy versus 90.13% for direct rank-4 training. Compressing rank 128 to rank 8 averaged 90.63% versus 90.30% for direct rank-8 training.

The operational rule is not “train at the highest rank available.” A rank-128 adapter compressed directly to rank 1 collapsed to 77.32%, compared with 90.08% for direct rank-1 training. Two recovery strategies matter: 200 additional target-rank steps raised that collapsed adapter to 90.05%, while gradual rank reduction during training reached 90.51% under the paper’s strongest 4B text schedule.

For teams managing many adapters, the practical opportunity is to treat rank as a two-stage decision: learn with enough capacity first, then choose the deployment footprint. The boundary is equally operational: compression severity must be validated, and the paper does not establish the same behavior outside standard LoRA or the Gemma 3 family.

A small deployment rank does not require a small training rank

LoRA constrains a task-specific weight update as

where the rank of the two factors determines the adapter’s capacity and size. In normal use, that rank is selected before fine-tuning. A team targeting rank 4 in production therefore trains a rank-4 adapter from the start.

That couples two different concerns. Training asks how much parameter space is useful for finding a strong task update. Deployment asks how much adapter state the serving system can afford to store, load, batch, and keep resident.

The paper’s central contribution is to separate them.

With Post-Squeeze, a higher-rank adapter is trained first. Its learned update is then decomposed and truncated to the strongest target-rank singular components before being refactored into a smaller LoRA. The final adapter has the deployment rank the operator wanted, but optimization did not have to remain inside that rank throughout training.

The detailed 13-task results in Appendix H support this distinction. Rank 16 compressed to 4 gained 0.28 average accuracy points over direct rank-4 training, while rank 128 compressed to 8 gained 0.33 points. These are modest improvements, not a universal dominance result: several individual tasks decline slightly. Their significance is architectural. A smaller final adapter need not have been trained exclusively as a small adapter.

More training capacity stops helping when compression removes the task update

The result creates an easy misconception: if higher-rank training gives optimization more room, start from the highest possible rank and compress afterward.

The paper’s aggressive-compression experiment shows why that rule fails.

For Gemma 3 4B across the 13 text tasks, direct rank-1 LoRA averaged 90.08%. A rank-128 adapter compressed immediately to rank 1 averaged only 77.32%. ANLI Round 2 fell to 29.30%, MMLU to 29.97%, and ARC-Challenge to 59.68%.

The paper examines this failure through singular-value energy retention:

Compression keeps the dominant singular directions of the learned update and discards the rest. Moderate reductions retain most of that squared singular-value mass: the paper reports roughly 96% or more when halving rank from 128 and roughly 94% or more when halving from 32 across the text tasks. Extreme reductions can remove substantially more.

This analysis is best read as a mechanism check, not an automatic deployment rule. The paper treats retained energy as a preliminary proxy for information loss; it does not validate a threshold that tells operators which source-to-target pair is safe.

Compression failure can be repaired without restarting the run

A collapsed adapter is not necessarily evidence that the target rank itself is inadequate. It may instead mean that truncation placed the adapter in a poor point within that low-rank space.

The paper tests this with Cont-Squeeze. After the 128-to-1 compression fell to 77.32%, another 200 fine-tuning steps at rank 1 raised the average to 90.05%. At 700 additional steps it reached 90.09%, essentially matching the original direct rank-1 result.

This experiment is a recovery test. It shows that information lost during a severe transformation can often be compensated by re-optimizing the compressed factors. It does not show that arbitrary compression is costless: continued tuning consumes additional training budget, and the paper does not supply a universal rule for when refinement will be cheaper than choosing a less aggressive compression ratio.

For an ML operations team deriving several deployment variants from one source adapter, however, it changes the fallback procedure. A bad compressed checkpoint need not immediately trigger a complete retraining job.

Gradual rank reduction is stronger than one hard squeeze

The paper also moves compression inside training.

In-Squeeze progressively lowers rank through a predetermined schedule—for example, 128 to 64 to 32 and onward—while keeping the overall training-step budget matched to the direct-training comparison. Instead of learning a high-rank solution and removing many directions at once, the model repeatedly adapts after smaller reductions.

On the main Gemma 3 4B text comparison, the standard schedule averaged 90.21%, while the minimum-steps schedule reached 90.51%, compared with 90.08% for direct rank-1 LoRA. The minimum-steps variant produced the best result on 8 of the 13 tasks.

That makes rank annealing the strongest overall strategy in the paper’s principal low-rank comparison. The useful interpretation is optimization-related: the model can exploit a more expressive space early and progressively accommodate the final constraint instead of confronting the smallest space from the first update.

The vision-language experiments provide a modality robustness check rather than a separate thesis. Across 10 tasks, direct rank-1 training averaged 85.20%. The standard In-Squeeze schedule reached 85.48%, while Cont-Squeeze after 200 recovery steps reached 85.51%. These results suggest the behavior is not confined to the text suite, although both settings still use Gemma 3 and standard LoRA.

The deployment value is adapter standardization after training

For platform teams serving many task adapters, the most consequential Cognaptus inference is not the few tenths of an accuracy point reported in moderate compression.

It is the possibility of separating adaptation search from serving format.

A team could tune a sufficiently expressive source adapter once, derive several smaller candidate ranks offline, validate them against its own quality threshold, and deploy a common rank across tasks that otherwise would have required heterogeneous adapter sizes. That could reduce repeated rank-specific training, simplify storage and loading, and make batching or adapter residency easier to manage.

The paper also removes one implementation obstacle. Straightforward compression can construct the full dense update $\Delta W$, which may be memory-intensive. Its QR-based formulation instead decomposes the original LoRA factors and performs the SVD on an $r_{src}\times r_{src}$ core matrix whose singular-value structure corresponds to the represented update. The reported implementation closely tracks standard Post-Squeeze without materializing the full model-sized delta.

For operators, that is an implementation contribution: the compression procedure itself does not have to erase the memory savings that made LoRA attractive.

What the paper does not yet make automatic

Three boundaries should remain attached to any production recommendation.

First, the experiments are concentrated on Gemma 3 1B, 4B, and 12B instruction-tuned models. The study provides useful model-size checks, but it does not establish equivalent behavior across unrelated model families.

Second, the method is evaluated on standard LoRA. Adaptive, structurally modified, or otherwise nonstandard LoRA variants may interact differently with rank transformation.

Third, the operator still has to choose the source rank, target rank, and annealing schedule. The energy-retention analysis explains why some transformations are dangerous, but it is not yet a validated controller for making those choices automatically. The smaller 1B model is also reported as more sensitive to aggressive compression, reinforcing that safe rank gaps may depend on model and task.

So the practical change is procedural, not universal: do not assume the rank required by the serving system must also define the entire optimization path. Train with enough room to learn the task update, compress to the footprint production requires, measure what the transformation removed, and retain a recovery path when the reduction is aggressive.

That turns LoRA rank from a single pre-training commitment into a sequence of operational decisions.

Cognaptus: Automate the Present, Incubate the Future.