Rationales Before Results: Teaching Multimodal LLMs to Actually Reason About Time Series

Dashboard work has a familiar little ritual.

Someone opens a chart, zooms into the last few points, notices a dip, a rebound, or a suspiciously clean trend line, and then says something that sounds analytical: “Looks like it will continue.” Sometimes that is wisdom. Sometimes it is just a human staring confidently at a squiggle.

Now replace the human with a multimodal large language model. The ritual becomes more expensive, more fluent, and not necessarily less squiggle-driven.

That is the problem behind RationaleTS, proposed in Rationale-Grounded In-Context Learning for Time Series Reasoning with Multimodal Large Language Models.¹ The paper’s central claim is simple but useful: multimodal LLMs may see time-series charts, but seeing is not the same as reasoning. If the model only receives a chart and a task, it may latch onto local visual patterns. If it receives ordinary examples, it may imitate surface similarity. If it receives chain-of-thought prompting, it may produce a nicer-looking explanation after making the same shallow guess. Wonderful. The spreadsheet has become poetic.

RationaleTS changes the object being retrieved. It does not retrieve raw examples, labels, or generic documents. It retrieves rationales: compact reasoning paths that connect observable temporal patterns to downstream implications.

That distinction is the paper’s most important idea. The benchmark gains matter, but the mechanism matters more.

The missing unit is not the chart, but the observation-to-implication bridge

Most business time-series work is not pure curve extrapolation. In finance, a one-day S&P 500 movement may depend on cross-market signals such as VIX, Nikkei 225, gold, oil, and exchange rates. In traffic, future occupancy depends not only on the last occupancy value, but on the interaction among intensity, humidity, wind speed, pollutants, and recent oscillation. In wind power, active power depends on meteorological variables, rotor behavior, pitch angles, and turbine state.

The paper frames this as time-series reasoning, not just forecasting. The target is usually a discrete future trend: increase, decrease, stable, or a binary future condition. The output should be supported by evidence from historical windows and temporal context.

This is exactly where chart-based multimodal LLMs become tempting. Turn the numbers into a chart. Let the model inspect it. Ask for a prediction and explanation. It feels natural because humans also reason from charts.

But the authors argue that current MLLM approaches often rely on “superficial temporal similarity or local pattern matching.” In practical terms, the model may see that the target variable fell in the last hour and assume the next change will remain similar. It can describe the chart, but it may fail to explain how other variables change the implication of that chart.

RationaleTS treats this failure as a missing-memory problem. The model needs not just examples, but reusable reasoning priors.

A rationale in this paper is not a post-hoc apology attached to an answer. It is a small, structured bridge:

Observation → Implication

For example, in the traffic case, a sharp collapse in intensity can imply flow breakdown and queuing; rising humidity under low wind speed can imply slower or more variable traffic; elevated NO2 can suggest vehicle-rich conditions; repeated troughs in occupancy can imply mean-reversion. The exact domain changes. The reasoning unit stays the same.

That is why the paper’s framing is more interesting than “RAG for time series.” RAG retrieves documents. ICL retrieves examples. RationaleTS retrieves ways of interpreting evidence.

RationaleTS works in three moves: generate, retrieve, infer

The method is easier to understand as a mechanism pipeline.

The first move is the subtle one.

During rationale-base construction, the system gives an MLLM the chart and the actual outcome. The model is asked to explain the outcome through concise causal-style reasoning paths. This is abductive generation: reasoning backward from known outcome to plausible evidence-based explanation.

But the paper includes an important guardrail. The generated rationale should not reveal the true label directly. The prompt asks for observation-to-implication bullets while avoiding explicit disclosure of the final outcome. This matters because the system is trying to build reusable reasoning logic, not a pile of answer keys wearing glasses.

At inference time, the query sample has no ground-truth label. So the system produces a factual text summary of the query chart, retrieves relevant rationales, and then asks the MLLM to analyze the new chart with those historical reasoning paths in context.

The result is a “rationales before results” pattern: the model is nudged to form evidence-grounded reasoning before producing the final classification.

Answer leakage is not intelligence, which apparently needed saying

One of the paper’s best design checks is the ablation around what gets included in the rationale base.

The authors test variants that incorporate the visual chart, the ground-truth label, or both into the rationale context. None improves over the proposed method. In fact, all three perform worse than RationaleTS on both Finance and Power.

This is a useful result for business AI design because it pushes against a very common instinct: “Just give the model everything.”

No. Sometimes “everything” is just a larger room for bad shortcuts.

If labels leak into rationales, the model may learn to associate retrieved text with final answers rather than transfer reasoning structure. If charts are packed into the retrieved context, the model may overweight local visual resemblance. RationaleTS works because the rationale base is deliberately constrained. It stores interpretation patterns, not full historical cases.

That is a small architectural choice with a large practical message: the quality of an AI memory system depends on what kind of object is allowed to become memory.

Hybrid retrieval matters because similar curves can mean different things

The second major contribution is retrieval design.

A naive time-series retrieval system would search for historical samples that look numerically similar. That helps, but it is not enough. A PM2.5 decline could come from wind conditions, seasonal effects, or measurement context. A traffic occupancy drop could mean smooth clearing traffic, or it could be a temporary trough before congestion rebounds. Same curve, different story. Time-series work is rude that way.

RationaleTS uses two retrieval signals.

The first is data-centric similarity. The method reorganizes raw time series into tabular form and uses TabPFN as a frozen encoder to obtain temporal embeddings. This signal captures statistical coordination among variables.

The second is semantic-centric similarity. Since the query sample has no ground-truth rationale, the system asks an MLLM to generate a factual summary of the chart. It then embeds that summary and compares it with rationale texts. This signal captures coarse semantic context: what seems to be happening, not only how the numbers move.

The final score combines the two with a balancing factor $\lambda$. In the paper’s implementation, the default is $K=5$ retrieved rationales and $\lambda=0.8$, meaning the retrieval leans heavily toward data-centric similarity while still retaining semantic context.

The ablation results show why both sides matter.

The Power result is especially sharp: removing data-centric similarity drops F1 from 71.50 to 52.50. That does not mean semantics are useless. It means semantics cannot replace the temporal structure of the data. The useful pattern is not “language beats numbers” or “numbers beat language.” The useful pattern is numbers select the right neighborhood; language helps interpret why that neighborhood matters.

The sensitivity analysis reinforces this. More retrieved rationales increase the chance that at least one rationale has the same label as the query, but more rationales do not guarantee better performance. The best setting is $K=5$. Beyond that, additional rationales can become noisy guidance. Retrieval quality beats retrieval quantity. We may now pause while every enterprise knowledge-base project pretends it already knew that.

The main results are not “MLLMs solve forecasting”

The paper evaluates RationaleTS on three domains:

• Finance: daily records from 2019 to 2023, with the task of classifying whether the next-day S&P 500 movement rises above 1%, falls below -1%, or remains stable after a 20-day window.
• Traffic: hourly records from weather and transportation indicators, with the task of classifying next-hour occupancy movement after a 12-hour window.
• Power: 10-minute wind turbine and meteorological records, with the task of deciding whether average active power in the next six hours exceeds that of the past 24 hours.

The authors compare against time-series reasoning methods, multiple MLLM baselines, textual and visual prompting, chain-of-thought, and standard in-context learning.

RationaleTS reports the best F1 and AUC across all three datasets.

The careful interpretation is not that MLLMs have become reliable forecasters. The tasks are classification-style trend reasoning, not open-ended trading, dispatch, or operational planning. The datasets are public benchmarks. The rationales are generated by a powerful model and not independently audited for correctness.

The better interpretation is narrower and more useful: when a multimodal model already has chart perception ability, giving it retrieved reasoning paths can improve trend-classification performance more reliably than simply changing modality, adding chain-of-thought, or retrieving ordinary examples.

That is the paper’s business-relevant message. It is not magic. It is scaffolding.

The sensitivity tests are about retrieval discipline

The paper’s sensitivity and efficiency sections should not be read as a second thesis. They are supporting tests around how the mechanism behaves.

The number of retrieved rationales, $K$, has a sweet spot. With too few rationales, the model lacks enough reference logic. With too many, irrelevant or contradictory rationales add noise. The reported optimum is $K=5$ on Finance and Power.

The balancing factor, $\lambda$, also matters. As $\lambda$ increases, the retrieval leans more toward data-centric similarity. The best setting is $\lambda=0.8$, but this does not correspond to the highest retrieval HitRate. That is important. A retrieval set can contain at least one same-label rationale and still fail to help reasoning if the rationale is semantically misleading or buried among noisy examples.

The efficiency analysis adds another useful warning. Variants that put visual charts into the prompt use much more input context: the paper reports 5.7 times more tokens compared with RationaleTS, while average F1 drops by 7.23 percentage points across Finance and Power. Including ground-truth labels also causes a reported average F1 drop of 4.17 points. VL-Time uses fewer tokens but gives lower average F1 than RationaleTS on the compared datasets.

The lesson is operational: context length is not free, and more context can reduce decision quality if it changes the model’s attention target.

For business systems, this matters because the usual production failure mode is not “we had too little context.” It is “we dumped too much poorly structured context into the prompt and called it retrieval.”

The case study shows the difference between explanation and chart narration

The Traffic case study is useful because it makes the mechanism visible.

The paper contrasts RationaleTS with a VL-Time-style reasoning process. The baseline focuses heavily on recent occupancy movement: hour 11 occupancy is 10, hour 12 is 8, the difference is -2, so the next change is predicted to fall into the same category. That is chart narration. It is clean, local, and not very deep.

RationaleTS, by contrast, builds a multi-step explanation around several variables and retrieved reasoning paths. It notices intensity and occupancy movement, humidity rising under low wind speed, NO2 increasing, and historical rebound patterns after sharp troughs. It then forms a conclusion about next-hour occupancy using both current observations and retrieved rationale patterns.

This does not prove causal truth. The rationales are still generated text. The dataset is still observational. The explanation can still be wrong.

But it does show the operational difference between two forms of model behavior:

The practical gain is not only accuracy. It is inspectability. A decision-support system that says “occupancy may rebound because prior similar troughs plus humidity and NO2 patterns implied slower flow” gives analysts something to challenge. A system that says “the last value went down, so maybe it stays down” gives them a horoscope with axis labels.

The business value is a rationale library, not a smarter dashboard

RationaleTS suggests a practical middle layer for business forecasting systems.

Many organizations already have dashboards. Some now have LLM wrappers on top of those dashboards. The missing layer is a curated memory of how domain experts connect observations to implications.

That memory could look like this:

For finance, this could help structure market-monitoring memos without pretending to be an autonomous trading brain. For energy operations, it could help turbine analysts compare current sensor conditions with prior reasoning patterns. For traffic management, it could support early warnings that explain which environmental and flow signals matter.

The ROI pathway is not “replace analysts.” That is the lazy deck-slide version. The more plausible pathway is:

1. reduce repeated chart interpretation work;
2. make model explanations easier to audit;
3. improve consistency across routine monitoring tasks;
4. create a reusable reasoning asset that grows with historical cases;
5. keep final decisions under human or rule-based governance.

This is less glamorous than “AI predicts the future.” It is also more likely to survive contact with an actual operations team.

Where the method can break in production

The paper is clear about one major limitation: RationaleTS depends heavily on generated reasoning paths, and those paths are not further evaluated. That is not a footnote-level issue. It is central.

If the rationale base contains weak, spurious, or domain-inappropriate reasoning, the retrieval system can faithfully retrieve bad guidance. The model may then produce a more coherent wrong answer. Coherence is useful. It is not the same as correctness. We have learned this lesson several thousand times and still keep scheduling refresher courses.

Several boundaries matter for business adoption.

First, the paper studies future trend classification tasks. It does not show that RationaleTS can handle continuous forecasting, long-horizon planning, anomaly diagnosis, or intervention design. Those may require different rationale structures.

Second, the datasets are public benchmarks. Production data is messier: missing values, sensor drift, regime shifts, non-stationarity, delayed labels, business-calendar effects, and silent process changes. Hybrid retrieval may help, but it does not remove the need for monitoring domain shift.

Third, the rationale generation process uses a powerful MLLM with ground-truth labels during base construction. That is reasonable for benchmark design, but in business deployment the cost and governance of generating and validating rationales must be counted.

Fourth, outcome categories can be coarse. Finance, for example, has a dominant “stable” class in the dataset details. A system can perform reasonably on aggregate metrics while still being weak on rare but business-critical movements. F1 and AUC are useful, but they are not a full risk report.

Finally, the method should be used as decision support. The paper itself notes risks around misuse in automated decision-making. That boundary is especially important in finance, healthcare, infrastructure, and safety-related operations.

The real lesson: teach the model what counts as reasoning

RationaleTS is valuable because it changes the question.

Instead of asking, “Can a multimodal LLM read a time-series chart?” it asks, “What kind of prior reasoning should the model retrieve before interpreting the chart?”

That is the better question.

Charts are inputs. Examples are inputs. Documents are inputs. But business reasoning often depends on something more specific: reusable links between observations and implications. When occupancy collapses while NO2 rises, when wind conditions shift while turbine output changes, when cross-market indicators diverge before an index move, the model needs more than visual perception. It needs structured memory of how such patterns have previously been interpreted.

The paper does not solve time-series reasoning in general. It does show a promising design pattern: build a rationale base, retrieve it with both numerical and semantic signals, and force the model to reason from those retrieved paths before producing a result.

That is a modest claim. Conveniently, modest claims are often the ones worth implementing.

Cognaptus: Automate the Present, Incubate the Future.