Dead Weights, Live Signals: When Frozen Models Start Talking

A model is usually treated like a finished machine. You send text in, get text out, and pretend the interesting part happens somewhere behind a curtain. If the answer is weak, the industry has a familiar menu: prompt harder, fine-tune, route to a bigger model, or pay the tax of yet another orchestration layer. Very elegant, in the way a pile of adapters behind a monitor is elegant.

The paper behind this article asks a more useful question: what if existing models are not finished machines, but frozen components that can still participate in a trainable system? In Dead Weights, Live Signals: Feedforward Graphs of Frozen Language Models, Marcus Armstrong, Navid Ayoobi, and Arjun Mukherjee propose a graph of heterogeneous frozen LLMs that communicate through learned projections into shared latent spaces, rather than through generated text or token-level voting.¹

That mechanism is the point. The headline numbers are attractive: 87.3% on ARC-Challenge, 82.8% on OpenBookQA, and 67.2% on MMLU, using only 17.6 million trainable parameters over roughly 12 billion frozen parameters. But the more interesting claim is not “small trainable layer beats some baselines.” We have seen that movie. The more interesting claim is that frozen model boundaries may not be hard walls. They may be interfaces.

The familiar mistake is treating multi-model AI as a text committee

Most business-facing multi-model systems today are committees with better branding. One model writes, another critiques, another routes, another summarizes the critique, and somewhere in the middle a manager gets a dashboard and a new invoice.

Technically, this style of system operates at the surface. Models exchange natural language, final answers, probability distributions, or routing decisions. That is useful, but it keeps each model as a black box. The internal representation of one model never becomes usable computation for another.

This paper moves the interface down one level.

Instead of asking five models to talk in text, it extracts hidden states from frozen models, projects those activations into a shared continuous space, injects the aggregate signal into other frozen models, and trains only the projection matrices plus a small output node. The LLMs themselves remain frozen. Their weights do not learn. The adapters between them do.

That distinction matters because it changes the economic object. A conventional ensemble uses models as vendors. This architecture uses models as infrastructure.

The paper is therefore less about model size and more about model plumbing. Less glamorous, yes. Also more likely to survive contact with enterprise budgets.

The graph is not an ensemble wearing a lab coat

The architecture has two layers of frozen LLM nodes and one trainable output node.

Layer 1 contains three smaller frozen models: Llama-3.2-1B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-2-2B-IT. Each receives the same input, but with a different analytical framing prefix. The paper uses framings such as factual content, reasoning structure, and language framing. Each model produces a hidden representation, extracted from a late layer and normalized.

Those representations are not decoded into answers. They are projected into a shared latent vector and averaged.

Layer 2 contains two larger frozen models: Phi-3-mini-4K-Instruct and Mistral-7B-Instruct-v0.3. The shared vector from Layer 1 is injected into their residual streams during the forward pass. The downstream models then continue computation under the influence of that injected signal.

Finally, a lightweight cross-attention output node aggregates the two Layer 2 representations and predicts one of four answer choices. This is why the paper’s benchmarks are multiple-choice tasks, not open-ended generation.

A simplified view looks like this:

Input question
   │
   ├── Frozen Llama-3.2-1B ─┐
   ├── Frozen Qwen2.5-1.5B ─┼── learned projections → shared latent signal
   └── Frozen Gemma-2-2B ───┘
                                      │
                       injected into residual streams
                                      │
                 ┌────────────────────┴────────────────────┐
                 │                                         │
        Frozen Phi-3-mini                         Frozen Mistral-7B
                 │                                         │
                 └──────── projected representations ──────┘
                                      │
                         cross-attention output node
                                      │
                              multiple-choice answer

The conceptual shift is small enough to state in one sentence and large enough to annoy a few architecture diagrams: the models do not collaborate by talking; they collaborate by having their internal states translated.

The learned components are only the projection matrices and the output node, totaling 17,580,036 trainable parameters, about 0.15% of the roughly 12 billion frozen parameters across the constituent models. The five frozen LLMs are therefore not being fine-tuned. They are being wired.

That is why the term “dead weights” in the paper title is doing actual work. The weights are frozen, but the system around them remains trainable.

Why residual-stream injection is the real mechanism

The paper builds on a premise from representation research: independently trained models may organize semantic information in geometrically compatible ways. They do not share the same coordinates, but they may encode related structures. If that premise is true, then a linear projection can act like a translation layer between models.

The graph uses that idea in a stronger form. It does not merely translate one model’s activation into another model for analysis. It trains projections end-to-end so the translated signal helps solve the downstream task.

This matters because the residual stream is not a decorative storage shelf. In transformer models, it is one of the main channels through which information is carried across layers. Injecting a vector into that stream means altering the computation of a frozen model without changing its weights.

For a business reader, the closest analogy is not “outsourcing a task to another expert.” It is closer to giving an expert a structured intermediate memo at the exact moment their reasoning process can still use it. The expert remains the same expert. The memo changes the trajectory.

The paper’s method has three operational consequences:

This is not the same as LoRA, prefix tuning, or a standard learned head on frozen representations. Those methods adapt a single model or classify a single model’s representation. Here, the learned part sits between models and trains the communication channel itself.

That is the paper’s main contribution. The benchmarks are evidence for whether the channel is useful, not the whole story.

The main results support communication, not just extra parameters

The paper evaluates the graph on three multiple-choice benchmarks: MMLU, ARC-Challenge, and OpenBookQA. These are not equivalent tasks. MMLU is broad subject knowledge across many domains. ARC-Challenge is structured science reasoning. OpenBookQA requires combining a core science fact with commonsense inference.

The comparison set includes two baselines. First, each constituent model is evaluated individually under a zero-shot greedy protocol. Second, the authors train a parameter-matched learned head on the frozen hidden states of the strongest single-model baseline for each benchmark. That second baseline is important. Without it, the graph might simply be winning because it has a trained classifier sitting on top.

The ARC result is the cleanest signal. The graph beats the best single model by 11.4 percentage points and the learned-head baseline by 9.1 points. On OpenBookQA, the graph also clears both comparisons by meaningful margins. On MMLU, the result is more modest against the best single model: +1.2 percentage points. The graph still beats the learned-head baseline by 6.7 points, but the best-single-model comparison should be read with restraint.

This pattern is useful. It suggests the architecture is strongest where the task benefits from combining complementary representations, especially structured reasoning and fact-integrated inference. It is less dramatic on broad subject recall, where a strong single model may already carry much of the necessary knowledge.

The paper’s training dynamics are also informative. ARC-Challenge reaches 80.7% after only 50 gradient steps. OpenBookQA reaches 74.0% within 30 steps, crosses the best single-model baseline during the first epoch, peaks at 85.4% mid-epoch 2, and ends at 82.8% test accuracy after epoch 3. MMLU converges more slowly and peaks at 67.2% around epoch 2.

Those curves are not a second thesis. They are implementation evidence: the graph is learnable, the gains are not arriving only after heroic training, and the easier-to-compose benchmarks show fast uptake. The industry should resist the temptation to turn that into “frozen model graphs will solve enterprise reasoning.” Please, for once, let a training curve remain a training curve.

The ablations ask whether the wires are actually carrying signal

The paper’s analysis section is more valuable than the raw benchmark table because it checks whether the architecture’s moving parts are doing what the architecture claims.

The gradient-flow validation is especially important. The authors build a minimal two-node graph from Llama-3.2-1B to Qwen2.5-1.5B, connected by one projection matrix and a four-class output head. A backward pass shows that the projection receives about 13% of the output head’s gradient signal.

That is attenuated. It is also not dead. For this paper, that difference is everything.

The authors also test a skip connection from the projected source representation directly to the output head. It provides no improvement in the reported gradient ratio at this depth. This is a useful negative result because it keeps the final architecture simpler. The result should not be overgeneralized. In deeper graphs, skip connections or auxiliary losses may become necessary. But for the paper’s two-layer architecture, the frozen boundary is not the fatal obstacle one might expect.

The projection-gradient analysis during MMLU training adds another diagnostic layer. All five projection matrices receive nonzero gradients throughout training. Layer 2 projections receive stronger gradients than Layer 1 projections, which is exactly what one would expect because they sit closer to the output node. The Layer 1 projections, however, do not meaningfully specialize; their gradient norms remain similar despite different framing prefixes.

That last finding is not a bug to hide in the appendix. It tells us where the system’s diversity is coming from. The benefit appears to come mainly from the heterogeneity of the frozen models, not from the learned projections becoming distinct experts. The adapters translate and carry signal. They do not yet become a rich division-of-labor system.

The routing result is interesting because it was not hand-coded

The output node develops asymmetric reliance on the two Layer 2 models. The paper reports a strong preference toward Phi-3-mini over Mistral-7B, especially early in training. On MMLU, Phi-3-mini-related gradient norms exceed Mistral-related norms by roughly 10–30 times early in training, later settling around 5–10 times by epoch 4. Mistral remains active, but secondary.

This result should be interpreted carefully. It does not mean Phi-3-mini is universally superior. It means that, inside this graph, on these tasks, under this training setup, the output node finds Phi-3-mini’s representations more useful or more geometrically compatible with the learned query. The authors connect this to prior claims about Phi-3-mini’s synthetic training corpus producing regular representations, but the paper does not isolate that causal mechanism with a controlled ablation.

Still, the result is worth attention. A manually designed router asks, “Which model should handle this query?” A differentiable graph asks a more internal question: “Which representation receives more useful gradient signal?”

That is a quieter kind of routing, but potentially a more powerful one. It is also harder to audit. A text router can be inspected through logs. A latent router leaves traces in attention weights, gradient norms, and representation geometry. Enterprise governance teams will love this. By “love,” I mean they will ask for six committees and a dashboard.

The business implication is composable AI, not free intelligence

The business interpretation should be separated into three layers: what the paper directly shows, what Cognaptus can reasonably infer, and what remains unresolved.

First, the paper directly shows that a small set of trainable projection layers can compose multiple frozen LLMs into a feedforward graph that beats selected single-model and learned-head baselines on three multiple-choice benchmarks. It also shows that gradients can pass through frozen model boundaries strongly enough to train those projections in a two-layer graph.

Second, the reasonable inference is that model portfolios may become more valuable if they can be connected below the text layer. Today, many companies think of model choice as a procurement question: which API, which model family, which vendor, which price tier. This paper points toward a different question: which frozen models should be wired together, at which layers, with what projection interfaces, for which class of tasks?

Third, unresolved questions remain large. The paper does not prove that this approach works for open-ended generation, long-context workflows, tool use, enterprise document analysis, compliance review, or production-grade agent systems. It does not test latency and deployment economics under real serving constraints. It does not show multi-run statistical confidence for every reported gain. It does not compare against every modern composition strategy.

So the practical takeaway is not “replace fine-tuning.” The practical takeaway is more precise: for some tasks, especially structured reasoning tasks with multiple-choice or classifier-style outputs, it may be possible to extract additional value from existing frozen models by training the interfaces between them.

That is a more boring sentence than “the future of AI is networks of frozen minds.” It is also much more useful.

For companies building AI systems, the paper is most relevant where models are locally hosted or open-weight, where internal activations are accessible, and where outputs can be formulated as classification, ranking, selection, or structured decision tasks. It is less immediately relevant for teams using only closed text APIs. You cannot inject into a residual stream through a pricing page. Tragic, but true.

The limitations are not footnotes; they define the use case

The authors are explicit about several limitations, and these should shape the business reading.

The first limitation is task format. The results are on multiple-choice benchmarks. The output node predicts among four answer choices. That is a long way from multi-turn advisory workflows, open-ended drafting, tool-using agents, or enterprise report generation. The architecture may extend, but the paper does not show it.

The second limitation is statistical confidence. The reported experiments use single training runs per benchmark. This matters most for MMLU, where the graph beats the best single model by only 1.2 percentage points. The larger ARC and OpenBookQA gains are harder to dismiss, but variance still matters.

The third limitation is Layer 1 non-specialization. The first-layer projections do not develop strong distinct roles. This means the current design depends heavily on the pretrained differences among the constituent models. If the model portfolio lacks useful diversity, the graph may have less to compose.

The fourth limitation is infrastructure. The full graph fits on a single 80GB A100 in the paper’s setup. That is feasible in research and some enterprise environments, but it is not the same as cheap CPU-side orchestration. Parameter efficiency in training does not automatically mean low serving cost.

The fifth limitation is schedule stability. The authors note instability from cosine annealing during checkpoint resumption and suggest warmup-then-constant schedules as a future improvement. This is not glamorous, but training recipes matter. Many promising architectures die quietly in the swamp between paper and reproducible engineering.

These limitations do not weaken the paper’s core idea. They locate it.

The next version should learn the first-layer mixture, not just average it

The most obvious technical improvement is also identified by the authors: replace fixed average pooling over Layer 1 projections with learned attention-weighted pooling. That would directly address the non-specialization problem.

Averaging is simple and stable, but it is blunt. If the factual-content encoder, reasoning-structure encoder, and language-framing encoder are all projected and then averaged equally, the system has limited incentive to let each pathway become distinct. A learned pooling mechanism could allow the graph to decide which upstream model matters more for each example.

That change would also make the architecture more business-relevant. Real enterprise tasks are not uniform. A compliance query, customer-support escalation, legal clause comparison, and technical troubleshooting question may need different model mixtures. A fixed average says, “Everyone gets one vote.” Learned pooling says, “The useful expert changes by case.”

Beyond that, future directions include warm-starting projections from offline Ridge mappings, scaling to deeper graphs, testing open-ended generation, and probing shared latent spaces for interpretability. The interpretability point is not cosmetic. If latent-space communication becomes a real interface layer, system builders will need tools to inspect what is being passed between models. Otherwise, the architecture becomes another black box built out of smaller black boxes, which is the sort of recursive joke only machine learning could monetize.

Frozen models may become inventory

The strategic reading is simple: model capability may increasingly come from composition, not only from training larger monoliths.

That does not mean frontier models stop mattering. It means the unit of engineering may shift. Instead of asking whether one model can do everything, system builders may ask how to assemble several frozen models whose internal representations can be aligned, injected, and routed.

For Cognaptus-style automation work, the interesting implication is not just cheaper training. It is cheaper experimentation with architectures. If frozen models become reusable inventory, then the differentiating asset becomes the interface layer: projection design, routing logic, task-specific graph topology, monitoring, and evaluation.

The paper does not prove that this future has arrived. It proves something narrower and more useful: frozen LLMs can be wired into a differentiable graph, trained through latent-space communication, and made to outperform relevant baselines on selected reasoning benchmarks.

That is enough to take seriously.

The models are frozen. The system is not.

Cognaptus: Automate the Present, Incubate the Future.