PEFT

A camera sees a plastic bottle, a dolphin, a car, or a suspicious object inside an X-ray scan. The business question is usually not philosophical. It is: can we adapt an existing vision model to this specific mess without retraining half the machine? That is where parameter-efficient fine-tuning sounds irresistible. Freeze most of the pretrained model. Add a small trainable module. Spend less money. Store fewer weights. Avoid turning every client dataset into a private bonfire of GPU time. Lovely. Procurement smiles. Engineers almost smile. ...

The adapter budget problem is not just training cost Budget is usually where fine-tuning conversations become less glamorous. A team wants a customized model. The engineer suggests LoRA because full fine-tuning is expensive. Everyone nods. Then the uncomfortable question arrives: which rank? A low rank is cheap but may underfit. A high rank may work better but costs more memory and inference compute. So the team trains several adapters, compares them, chooses one, and pretends the search process was a minor detail. It was not. It was the hidden invoice. ...

Adapters are supposed to make fine-tuning simple. A team takes a large pretrained model, freezes most of it, trains a small adapter for customer support, another for invoice extraction, another for compliance review, and so on. The pitch is attractive: less storage, less training cost, faster iteration, fewer excuses from the infrastructure team. Naturally, the adapter becomes the small and tidy object everyone wants to manage. ...

Opening — Why this matters now Enterprise AI is entering its less glamorous, more useful phase: not “Can we connect an LLM to everything?” but “Can we adapt it without making the GPU bill look like a small infrastructure project?” Fine-tuning still matters. Retrieval helps with knowledge access, prompt engineering helps with behavior shaping, and agent frameworks help with workflow orchestration. But many businesses eventually hit the same wall: the base model is close, yet not close enough. It needs domain style, task format, compliance habits, tool-use discipline, or workflow-specific judgment. That usually means some form of supervised fine-tuning. ...

Opening — Why this matters now Enterprise AI is entering its less glamorous phase: not the demo, not the keynote, not the charming chatbot that answers three curated questions correctly, but the operational grind of making models behave reliably inside messy workflows. That grind usually runs into a familiar triangle. Full fine-tuning is powerful but expensive, operationally heavy, and often risky when the training set is narrow. Parameter-efficient fine-tuning, especially LoRA-style adaptation, is cheaper and easier to deploy, but the smallest adapters can hit a ceiling. Meanwhile, the business user does not care whether the adapter was elegant. They care whether the model stops making the same costly mistakes in invoicing, compliance review, customer support, code generation, or scientific triage. ...

Opening — Why this matters now Fine-tuning large models used to sound like a research luxury. Now it is a line item in the infrastructure budget. Enterprises do not want one general-purpose model behaving vaguely usefully for everyone. They want domain-specific behavior: a support adapter for insurance claims, a compliance adapter for legal review, a financial-document adapter for analyst workflows, perhaps a dozen regional variants, and then another dozen because someone discovered “brand tone” during a steering committee meeting. Naturally. ...

Updating a model sounds harmless until the old parts of the system start reading the new representations incorrectly. That is the less theatrical version of catastrophic forgetting. Not the dramatic story where a neural network “forgets everything” like a distracted intern. The more useful story is quieter: a deployed AI system adapts its backbone to new data, the feature space shifts, and classifiers trained for earlier tasks are left calibrated to yesterday’s geometry. ...

More Rank Is Not Always More Capacity Fine-tuning teams love a simple knob. If the model underperforms, increase rank. If the adapter looks too small, increase rank. If the downstream task is hard, increase rank again and call it strategy. This is comforting because rank is measurable, budgetable, and easy to explain in a meeting. Unfortunately, reality has its usual habit of being less cooperative. ...

Formulation is where AI enthusiasm usually goes to sober up. In a slide deck, “AI-assisted drug development” sounds clean: feed the model a drug, get back a formulation, reduce experiments, accelerate personalisation, everybody nods. In a lab, the problem is less polite. A formulation is not just a sentence with chemical names. It is a physical recipe with roles, proportions, processing constraints, and mechanical consequences. A model can sound fluent while quietly omitting the lubricant, mangling the unit, or inventing a polymer that belongs more to fantasy literature than pharmaceutics. ...