Model Adaptation

TL;DR for operators AI systems do not only fail because they are too small, too dumb, or insufficiently blessed by the gods of scale. They often fail because the formal objective supervises one slice of behavior and quietly leaves another slice unmanaged. Three recent papers make that point from different domains. MA-SBI shows how side-channel context can correct simulation-based inference when the simulator is misspecified.1 A paper on non-adversarial LLM robustness shows that semantically neutral prompt changes can systematically shift internal module outputs, and that targeted debiasing can recover robustness without full retraining.2 FiberTune shows that robot policy fine-tuning can preserve action-equivalent visual residuals that ordinary action loss is happy to compress into oblivion.3 ...

Adapters are supposed to make fine-tuning simple. A team takes a large pretrained model, freezes most of it, trains a small adapter for customer support, another for invoice extraction, another for compliance review, and so on. The pitch is attractive: less storage, less training cost, faster iteration, fewer excuses from the infrastructure team. Naturally, the adapter becomes the small and tidy object everyone wants to manage. ...

Inspection cameras, pathology scanners, product catalog systems, and retail shelf analytics all create the same inconvenient problem: the image may look simple, but the knowledge needed to interpret it rarely comes from one source. A model trained on broad natural images may recognize general objects well. A contrastive model may separate fine visual categories better. A medical encoder may notice domain-specific patterns that a general model treats as visual noise. A segmentation-oriented model may understand spatial boundaries better than a classifier. Asking one backbone to cover all of this is elegant in a slide deck and occasionally foolish in production. Nature, sadly, did not optimize itself for clean model procurement. ...

Long prompts are easy to understand. They are also expensive, slow, and—in multimodal systems—very quickly ridiculous. That is the practical tension behind many-shot multimodal in-context learning. In principle, giving a vision-language model more examples should help it recognise the task. In practice, every image costs tokens, every additional demonstration adds latency, and open-source large multimodal models do not generally enjoy infinite context windows. The business version of the problem is familiar: you want a model to adapt to a specialised workflow, but you do not want to fine-tune it every week, pay for swollen prompts forever, or discover that the “cheap” approach now requires a larger GPU. ...