More Math, Less Finance: The Checkpoint Trade-off

TL;DR for operators

A financial AI model can become substantially better at mathematical reasoning while losing financial knowledge acquired during earlier training. In one Japanese financial-model pipeline, reasoning training raised mathematical accuracy by 9.5 percentage points but reduced financial knowledge accuracy by 21.9 points. Subsequent financial training recovered much of the loss without fully restoring the earlier score.

The deployment decision therefore depends on the workload. Skipping the reasoning stage produced the strongest financial knowledge results, but considerably weaker mathematical performance. An apparent improvement in fraud detection also failed to remain the strongest result under repeated evaluation.

For teams developing specialized financial assistants, the evidence favors preserving intermediate checkpoints, measuring numerical reasoning and financial knowledge separately, and validating the intended task mix before deployment. Crucially, although the model was trained using document images, the reported benchmarks used text inputs. Its visual document-reading performance remains unestablished.

A stronger financial model can forget finance

Consider a financial AI team developing a model to interpret disclosures, answer regulatory questions, and perform numerical analysis. The team first teaches the model to work with financial documents, then strengthens its mathematical reasoning, and finally gives it additional financial training.

The expectation is that the completed model should combine the benefits of all three stages. Yet the intermediate reasoning stage can improve calculations while reducing the model’s ability to answer financial knowledge questions. The final financial stage may recover some of that lost capability without producing the strongest financial knowledge model.

This is the deployment problem examined by Genshin Kakimoto and Atsushi Yanagisawa in Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial Documents.¹

Their approximately 8.4-billion-parameter model combines a SigLIP-v2 vision encoder, a projection module connecting visual representations to language, and the Japanese language model llm-jp-4-8B-Instruct.

Training proceeds through three phases:

• P1 — Vision-language alignment: Japanese image captions, OCR, and visual question-answering data, including synthetic examples derived from government PDFs.
• P2 — Mathematical reasoning distillation: Approximately 150,000 sampled mathematical problems with structured reasoning demonstrations generated by a teacher model, used to fine-tune the language backbone through QLoRA.
• P3 — Financial-domain adaptation: Financial QA training followed by synthetic financial visual QA, using LoRA-based adaptation.

This sequence makes it possible to compare what successive training phases contribute and what previously acquired capabilities they may disrupt.

The reasoning gain comes with a larger financial knowledge loss

The principal checkpoint evaluation reveals sharply different trajectories for mathematical reasoning and Japanese financial knowledge.

Source: Section 3, Table 2. Results use greedy decoding. Both columns report accuracy, but on different tasks.

From P1 to P2, GSM8K accuracy increases by 9.5 percentage points. Over the same transition, JP Fin Harness accuracy declines by 21.9 points.

This is the study’s central evidence of non-monotonic adaptation: improving one capability coincides with substantial deterioration in another.

P3 produces a partial reversal. Financial knowledge accuracy rises from 31.6% to 49.6%, while mathematical accuracy remains relatively strong at 71.6%.

However, the financial knowledge score remains 3.9 points below P1. Much of P3’s apparent gain represents recovery from deterioration introduced during P2, rather than improvement beyond the earlier checkpoint.

The task-level results reinforce this interpretation. On JP Fin Harness’s chabsa subtask, accuracy falls from 94.28% at P1 to 58.14% at P2, then recovers to 93.53% after the final P3 sub-stage.

The authors interpret these patterns as possible asymmetric interference: mathematical problem-solving capability persists relatively well through subsequent financial training, while previously acquired domain knowledge appears more vulnerable to reasoning-focused adaptation.

That interpretation is plausible within this pipeline, but the results do not establish that procedural reasoning is universally more resistant to forgetting than declarative knowledge.

Removing the reasoning phase exposes the checkpoint trade-off

The researchers also conduct a skip-phase ablation: they omit P2 and train the model through P1 followed directly by P3.

This comparison tests whether the intermediate reasoning stage contributes enough to justify its effects on other capabilities.

The answer depends on the benchmark.

Without P2, JP Fin Harness accuracy reaches 56.2%, compared with 49.6% for the complete P3 model. EDINET industry classification accuracy also increases from 8.2% to 10.4%.

But GSM8K accuracy falls from 71.6% to 59.6%.

Thus, skipping reasoning distillation gains 6.6 percentage points in financial knowledge accuracy while sacrificing 12.0 points in mathematical reasoning relative to the completed model.

The appendix provides a further qualification: the skip-P2 model leads four of five JP Fin Harness subtasks, while the unadapted P0 checkpoint leads the remaining cpa_audit task.

No single checkpoint dominates every evaluated capability.

For a financial research assistant whose primary workload involves Japanese regulatory and financial knowledge questions, the skip-P2 pathway may deserve priority in deployment testing. For a system that must solve multi-step numerical problems, the complete P3 pathway offers stronger mathematical benchmark performance.

These are task-specific selection hypotheses, not proof that either pathway performs better on an entire production workflow.

The strongest fraud result does not survive the robustness comparison

Fraud detection initially appears to offer a possible exception to the trade-off.

In the main single-run results, P3 achieves an EDINET fraud ROC-AUC of 0.614, compared with 0.554 at P1 and 0.499 when P2 is omitted.

One interpretation is that mathematical reasoning training enables stronger fraud detection after subsequent domain adaptation.

The robustness evaluation weakens that interpretation.

Across three decoding seeds, the mean fraud ROC-AUC is 0.562 at P1, 0.523 at P2, and 0.547 at P3. The highest mean now belongs to P1, not the final checkpoint.

These repeated evaluations serve as a robustness check, rather than an independent test of the proposed enabling mechanism. They also use sampling-based decoding instead of the greedy decoding applied in the principal comparison.

Importantly, the skip-P2 checkpoint has no corresponding three-seed evaluation. The evidence therefore cannot reliably establish whether retaining mathematical reasoning distillation improves fraud detection.

The paper also places the best single-run P3 fraud result near a previously published GPT-5 score of 0.62. That contextual comparison combines results from different evaluation sources and run settings; it is not a controlled head-to-head experiment.

For a financial risk team considering model selection based on fraud ROC-AUC, the relevant lesson is to require repeated, matched evaluation before treating a narrow score advantage as dependable. EDINET earnings classification, meanwhile, remains near chance across checkpoints, offering no convincing evidence of capability improvement.

Training on document images is not validation of document reading

The architecture and training data are multimodal. The evaluation evidence is narrower.

All five primary reported benchmark outcomes use text inputs. The experiments consequently measure changes in the language backbone’s financial and mathematical capabilities after multimodal training, rather than directly testing image-based document comprehension.

That distinction matters for document automation.

A team deploying the model to process Japanese financial disclosures would still need separate tests for page-image transcription, table extraction, layout interpretation, and image-grounded answers. Training on OCR and visual QA examples does not establish the accuracy of these operations at inference time.

The study also uses single-page documents during training. Multi-page understanding is not evaluated.

The paper supports conclusions about staged adaptation and text-input capability retention. It does not establish production-ready visual financial-document processing.

Checkpoint selection should follow the workflow

The findings support several bounded design choices for specialized financial AI development.

Preserve intermediate checkpoints. When mathematical reasoning and financial knowledge peak at different training stages, keeping P1, P2, and P3 available preserves options for subsequent task-specific evaluation.

Measure capabilities independently. Financial knowledge, numerical reasoning, classification, and image-grounded extraction should have separate acceptance criteria. A combined score can conceal a deterioration that matters to a particular financial workflow.

Evaluate alternative training schedules. Replay, data interleaving, and parameter-isolated adapters are plausible approaches to reducing interference, but this study does not experimentally validate them.

These choices also require attention to unresolved experimental confounding. Training phases differ in corpus composition, example counts, optimization settings, and adapter methods. P2 uses four-bit QLoRA, while other adapter-based stages use full-precision LoRA. The results cannot independently attribute each capability change to the training objective, phase order, or quantization.

The evidence comes from one Japanese financial-model architecture. Whether the same trade-offs occur with other model families, languages, or financial tasks remains an empirical question.

The final checkpoint is a candidate, not a conclusion

The study demonstrates that successive adaptation can produce meaningful but conflicting capability changes. Mathematical reasoning improves, financial knowledge deteriorates, later domain training partially restores it, and omitting an intermediate stage changes which tasks the model performs best.

For teams developing specialized financial AI, the consequential decision is not simply how many training phases to complete. It is which capabilities the intended workflow requires, which checkpoint best preserves them, and whether the measured advantage survives evaluation under relevant conditions.

A completed training sequence provides a new model checkpoint. It does not, by itself, establish the best model to deploy.

Cognaptus: Automate the Present, Incubate the Future.