Small Backbone, Large Memory: What CHARM Actually Improves

TL;DR for operators

A small neural backbone is not necessarily a small learned system: auxiliary task memories also consume parameters, and multiple inference attempts consume compute. In Structured Sparse Memory for Recurrent Reasoning¹, CoSE cuts task-memory parameters by about 94% while improving controlled ARC accuracy. But CHARM’s strongest results also rely on rule-preserving synthetic examples, carefully configured recurrence, longer training, and inference-time aggregation. For teams evaluating specialized reasoning models, the actionable comparison is task accuracy versus total learned footprint, training-data investment, and inference cost. Private-memory updates are promising for adding related tasks without changing shared weights; the experiments do not demonstrate production savings or reliable generalization to arbitrary new work.

The parameter count stops at the wrong boundary

A team choosing a compact reasoning system can easily compare the size of the network that performs its iterative computation. That number is incomplete if the network also consults learned vectors representing particular tasks. Those vectors are model parameters too, even when stored outside the main computation module.

Zixuan Zhao and colleagues examine this accounting gap in recurrent ARC architectures. The architecture they study has a roughly 14-million-parameter reasoning backbone, but its conventional ARC-AGI-1 task table contains approximately 448 million learned parameters: 875,617 entries, each 512 dimensions wide. An ARC-AGI-2 table reaches roughly 610 million. Reporting the backbone alone therefore obscures most of the baseline system’s learned capacity (Section 3.1; Appendix A.2).

The engineering question is how much of that table represents genuinely independent task information. ARC puzzles ask a model to infer transformations of small colored grids. During training, a puzzle can appear with different geometric transformations and color permutations. The conventional table assigns a separate vector to each puzzle–transformation combination, although many combinations share identifiable structure.

CoSE shares the regularities and stores the exceptions

The paper’s Compositional Sparse Embedding (CoSE) constructs a task representation from reusable factors: puzzle identity, geometric transformation, and color permutation. Learned scaling and shifting operations combine these factors into a full-width conditioning vector. CoSE then adds a narrow, instance-specific residual—typically 32 dimensions projected into a 512-dimensional embedding—to retain information that factor sharing alone cannot capture. Some variants gate this correction conditionally (Sections 3.2–3.3).

This combination matters more than either compression technique alone. In the matched ARC-AGI-1 memory ablation, with the backbone, Re-ARC synthetic data, and optimization unchanged, a full table uses 448.9 million embedding parameters and scores 78.50% pass@2. Rank-32 CoSE uses 29.6 million and scores 80.71%: about 93.4% fewer memory parameters and a 2.21-percentage-point gain. Pass@2 credits a puzzle when at least one of up to two exact-output attempts succeeds. Reported repeated-run results are 78.50% ± 1.03 and 80.71% ± 0.51 (Section 3.3, Table 4).

The ablation also shows why the design is not simply low-rank compression. A rank-32 lookup alone reaches 76.62% pass@2; a composition-only representation reaches 79.25%, but falls behind the full table on pass@1000. CoSE improves the two-attempt result, although its own 90.62% pass@1000 remains below the full table’s 91.38%. Different inference budgets expose different capacity trade-offs.

Crucially, the approximately 94% reduction applies to task-memory parameters, not all parameters, memory traffic, or runtime. The integrated ARC-AGI-1 CHARM system still contains approximately 44 million learned parameters, including its backbone.

Better memory is only one contributor to CHARM’s accuracy

The paper’s ARC-AGI-2 component comparisons isolate a second resource: task-matched synthetic training examples. With no synthetic data, full-table and CoSE systems obtain 15.42% and 18.61% pass@2, respectively. With Re-ARC2 data, the corresponding scores rise to 28.23% and 37.22%. At matched backbone, optimization, and evaluation settings, adding Re-ARC2 to CoSE is associated with an 18.61-point increase, considerably larger than the 3.19-point difference between memory architectures without that data (Section 4.1, Table 7).

Re-ARC2 is not an effortless data multiplier. It covers all 1,000 ARC-AGI-2 training tasks: 391 already had Re-ARC coverage and 609 required newly generated programs. Generation and rule verification were supplemented by human review and an independent audit; 19 generators required initial corrections, and seven second-round audit flags exposed genuine synthetic-data problems (Appendix D). The result is evidence for a generation–verification–review workflow, not proof that agreement between automated checks guarantees the intended puzzle rule.

CHARM integrates CoSE memory with a recurrent reasoning backbone, synthetic augmentation, and aggregation across augmented inputs, checkpoints, or refinement depths. Recurrent-depth studies test computation allocation and training credit assignment: more repeated steps do not consistently help unless unique-layer capacity and the backpropagation horizon are balanced (Section 4.2; Appendix E). Checkpoint and depth aggregation improve some standard-budget results, with smaller benefits after extended training (Appendix F). Both choices complicate latency and compute comparisons.

With extended training, the complete system records 84.0% pass@2 on ARC-AGI-1 and 46.7% on ARC-AGI-2 public evaluation (Section 4, Table 5). These are full-system results, not estimates of CoSE’s isolated effect; the main memory and synthetic-data ablations use shorter, specified training budgets.

Private memory enables adaptation, but not general transfer

For a deployed system that encounters additional related tasks, CoSE suggests an alternative to repeatedly updating shared neural weights: freeze the backbone and learn new task-private memory rows. On 233 held-out ARC-AGI-2 training puzzles, rank-32 memory-only adaptation achieves 43.58% pass@2, compared with 19.73% for full-table memory-only adaptation. Composition-only adaptation reaches 39.73% while updating just 512 parameters per puzzle. On the separate 114-puzzle public-evaluation adaptation stream, rank-32 CoSE reaches a much lower 12.19% (Appendix C.2, Table 26).

In sequential-learning tests, private-row updates preserve 80.00% ARC-AGI-1 pass@2 from the base checkpoint, while naive updates to the entire shared model drive that retained score to zero. Independent per-puzzle full fine-tuning can score higher on immediate adaptation, but it does not maintain one common model across tasks (Appendix C.3, Table 28).

The supplementary cellular-automata experiments reveal a limit to compositional transfer. CoSE performs best on observed rule–horizon combinations at 58.86% pass@2, yet composition-only memory does better on withheld combinations: 3.69%, versus 2.73% for CoSE and 0% for the full table. The residual helps fit observed combinations without ensuring reliable extrapolation. Small nanochat experiments also find lower validation bits per byte than Engram memory at tested nonzero capacities, but the gain narrows with more training data (Appendix B). These are exploratory extensions, not evidence of general-purpose task transfer.

What an architecture review should measure

Cognaptus inference: For teams selecting a specialized reasoning architecture with repeatable, decomposable task identities, evaluation should count backbone weights and auxiliary learned memory; compare task accuracy at a fixed inference budget; and report the cost of generating, checking, and maintaining training examples. For continual customization, compare frozen-weight private-memory updates with shared-weight fine-tuning on both new-task accuracy and old-task retention. These are testable evaluation requirements, not measured enterprise return on investment.

Boundary: The main ARC experiments use an XRM protocol that includes demonstration pairs from public-evaluation puzzles in training, though not their held-out answers. It is therefore not a strictly unseen-task test. Some architectural variants have only one run, and the integrated scores change multiple system components at once. The study has not established real-world latency savings, deployment reliability, or performance on broad, unfamiliar business tasks (Sections 6–7).

CHARM’s strongest architectural result is that task information need not always be stored as a separate full-size vector. Its strongest system-level lesson is equally concrete: model performance reflects the joint allocation of memory, training data, iterative computation, and inference budget. A compact backbone is only one line of that accounting.

Cognaptus: Automate the Present, Incubate the Future.