LoRA’s Missing Budget: Which Matrices Deserve an Adapter?
TL;DR for operators LoRA already avoids the cost of updating an entire pretrained model, but it can still spend adapter capacity uniformly across matrices that do not appear equally responsive to low-rank changes. If a training team has a fixed fine-tuning budget, there are therefore two allocation decisions: how large each adapter should be, and which matrices should receive one. ...