When Text Doesn’t Help: Rethinking Multimodality in Forecasting

TL;DR for operators

Text does not automatically make forecasts smarter. It often just makes the pipeline heavier.

A new AWS study benchmarks multimodal time-series forecasting across 16 datasets and 7 domains, comparing time-series-only models, alignment-based multimodal models, and direct LLM prompting.¹ The uncomfortable result is that multimodality is not a universal upgrade. Strong unimodal models still win on a substantial share of the benchmark, and the paper’s statistical tests do not support a blanket claim that adding text reliably improves accuracy.

The practical rule is simple: before adding reports, notes, product descriptions, weather commentary, market events, or customer narratives to a forecasting system, ask whether that text contains information the historical time series cannot already reveal. If the text merely says “sales have been rising” while the sales curve is already rising, congratulations, you have built a very expensive echo.

For production forecasting, alignment-based methods look more credible than direct prompting when accuracy is the objective. Direct LLM prompting can be useful in low-data exploration, rapid prototyping, or analyst workflows, but it remains weaker than dedicated forecasting backbones in the paper’s main tests. Bigger and reasoning-oriented LLMs help in some cases, but they do not magically become temporal statisticians because someone wrapped the data in XML tags and whispered “think step by step”.

The business takeaway is not “ignore text”. It is sharper: text is valuable when it is complementary, forward-looking, and technically aligned with the time-series representation. Otherwise, it is another feature ingestion cost wearing a lab coat.

The easy mistake is treating text as context, not as evidence

Forecasting meetings have a familiar rhythm. Sales are volatile, demand is drifting, energy usage is misbehaving, hospital load is hard to predict, or traffic flow refuses to respect the spreadsheet. Someone says, reasonably enough, “But we have text. We have product descriptions, weather reports, clinical notes, news summaries, maintenance logs, call-centre notes, campaign calendars. Surely the model should use that.”

The intuition is not foolish. Text can contain causal clues. A clinical note may describe deterioration before vitals show the full pattern. A weather report may explain energy demand. A product description may distinguish a seasonal coat from a basic T-shirt. A market note may flag a policy shock before the numbers settle into their new regime.

The mistake is assuming that because text is human-readable, it is automatically forecast-useful. Forecasting is not impressed by narrative richness. It wants information that changes the conditional distribution of the future.

That is the useful discipline of this paper. It does not ask whether multimodal forecasting is fashionable. That would be a short paper, mostly screenshots of conference agendas. It asks when multimodal time-series forecasting actually improves predictive performance, and what conditions make those gains appear.

The answer is conditional, occasionally counterintuitive, and commercially important.

The benchmark surprise is the main evidence

The paper evaluates two broad ways of combining text with time series.

The first is alignment-based multimodal forecasting. Here, a time-series model encodes the numerical history, a text model encodes the associated language, and a fusion mechanism combines the representations before forecasting. This is the engineered route: less glamorous than asking a giant chatbot to forecast, but more compatible with production modelling.

The second is prompting-based forecasting. Here, the time series and text are formatted into a prompt, and a large language model generates forecast values directly. This is the route that looks impressive in a demo and makes forecasting teams instinctively reach for a fire extinguisher.

The paper tests these paradigms against unimodal baselines including models such as PatchTST, DLinear, and Chronos. The benchmark covers dynamic and static text settings across domains including agriculture, climate, economy, energy, environment, health, social good, traffic, fashion, weather, medical data, finance, electricity, and solar generation.

The headline finding: multimodal models do not consistently outperform the strongest time-series-only baselines.

In the main table, unimodal models are best on 6 of the 16 datasets by MSE and 7 of the 16 by MAE. A paired Student’s t-test comparing unimodal Chronos with alignment-based Chronos gives p-values of 0.789 for MSE and 0.247 for MAE. In plain English: the paper does not find a statistically convincing overall improvement from adding text in that comparison.

That is the first useful correction. The right baseline for multimodal forecasting is not a weak time-series model. It is the strongest practical time-series-only model you can deploy. If text cannot beat that, it is not “context”. It is overhead.

Alignment usually beats prompting, but neither gets a free pass

The paper also finds that alignment-based models tend to perform better than prompting-based ones. Across the main benchmark, alignment-based approaches achieve the lowest MSE on 12 datasets and the lowest MAE on 9 datasets. Prompting-based methods achieve the lowest MSE on 3 datasets and the lowest MAE on 2 datasets.

That distinction matters for operators.

Prompting an LLM with a time series and a paragraph of context is attractive because it is easy to prototype. It also matches how analysts talk: “Here is the history, here is the context, tell me what happens next.” The problem is that current LLMs are still not reliably precise numerical extrapolators. They can describe patterns, infer plausible stories, and generate structured outputs. They are less dependable when asked to preserve scale, seasonality, variance, and temporal dynamics across many forecasting tasks.

Alignment-based systems have a less theatrical advantage. They preserve a dedicated time-series backbone and use text as an auxiliary signal. The architecture is allowed to respect the fact that time series are not essays with timestamps.

That does not make alignment-based multimodality automatically superior. It makes it the more serious default when the goal is forecast accuracy rather than conversational convenience.

What each experiment is really doing

The paper’s value comes from separating main evidence from diagnostic evidence. Not every table is trying to prove the same thing.

This matters because the paper is easy to misread. A casual reader may leave with “multimodality failed”. That is too crude. The actual result is more operational: multimodality behaves like any other expensive modelling choice. It helps under identifiable conditions, and it disappoints when those conditions are absent.

Text helps weaker time-series models because it fills gaps, not because it is magic

One of the paper’s clearer mechanisms is that text helps more when the underlying time-series model is weaker.

This should not be surprising, though it is often ignored. A strong time-series model already captures trend, seasonality, local dynamics, volatility patterns, and recurrent structures. If the auxiliary text merely restates those patterns, it has little left to contribute. A weaker model, by contrast, may miss temporal structure that text can summarise in a more accessible way.

That creates an uncomfortable procurement question. If your multimodal system beats a weak baseline, have you built a useful multimodal forecaster, or have you merely patched a bad forecasting backbone with text?

The answer matters for cost. Text pipelines require ingestion, cleaning, matching, embedding, storage, governance, and monitoring. If a stronger time-series-only model can achieve the same or better accuracy, the cheaper solution may be to improve the numeric forecaster first. Less glamorous, yes. Also less likely to require a post-mortem.

Fusion design is not plumbing

The paper’s alignment experiments examine choices that can look like engineering details: addition versus concatenation, average pooling versus CLS pooling, residual projection versus MLP projection, early versus middle versus late fusion, and different fine-tuning strategies.

These are not decorative knobs. They change performance.

The paper finds that some choices perform consistently well on average: addition, average pooling, residual projection, late fusion, and efficient fine-tuning. The reasons are practical. Addition avoids feature explosion and overfitting. Average pooling captures broader textual context better than relying on a single token. Residual projection helps preserve temporal information. Late fusion reduces the risk of disrupting the time-series encoder before it has done its job.

This is where enterprise implementations often go wrong. The team says “we added text embeddings”, as if the embeddings were seasoning. But where and how the text enters the model affects whether it provides signal or simply interferes with temporal representation learning.

A useful mental model: the text should advise the forecast, not trample the time-series encoder on the way in.

The data requirement is the finding executives will least enjoy

A popular argument for multimodality is that text should help especially in low-data settings. When numeric history is thin, surely language can compensate.

The paper complicates that story. For alignment-based multimodal forecasting, gains are more likely when there is sufficient training data. The authors observe a positive relationship between dataset size and multimodal improvement across their real-world benchmark.

The mechanism is straightforward. Alignment-based multimodal models need to learn cross-modal relationships. That is not free. If there are too few examples, the model may not learn whether the text is predictive, redundant, noisy, or misleading. It may simply learn a fragile association that looks useful until the next reporting cycle.

There is a narrow exception. Prompting-based methods can be useful in data-scarce or zero-shot settings because they do not require the same supervised alignment training. But this is a different use case. It is closer to exploratory forecasting support than high-confidence production modelling.

For operators, the point is blunt: if you do not have enough paired time-series and text examples, do not assume multimodality will rescue you. It may just give your sparse data a more expensive failure mode.

The synthetic tests reveal the real condition: complementarity

The most clarifying part of the paper is the controlled synthetic experiment.

The authors construct time series using Gaussian processes and explicitly control whether the text contains information that is unique or redundant. In the “unique” condition, the text provides information about a future change that is not visible in the observed time-series context. In the “redundant” condition, the same kind of information is already inferable from the historical series.

The result is exactly what a sober forecasting team should want to see: multimodality helps when the text contains unique predictive information. It offers little meaningful advantage when the text is redundant.

The numerical contrast is especially clear in the synthetic trend setting. With unique text, the unimodal model records MSE/MAE of 0.811/0.677, while the multimodal model improves to 0.477/0.408. With redundant text, the difference is much smaller: 0.449/0.394 for unimodal versus 0.428/0.376 for multimodal. In the seasonality setting, unique text produces an even sharper MSE improvement, while redundant text can slightly worsen performance. In the spike setting, the gain is modest even when text is unique.

This is the paper’s most business-relevant mechanism. Text is valuable not because it is text, but because it may encode events, constraints, plans, or signals not present in the numerical history.

That distinction changes how teams should evaluate data sources.

A product description may help retail demand forecasting if it encodes category, seasonality, style, or lifecycle stage not obvious from early sales. It may not help if years of sales history already reveal the pattern.

A weather report may help energy forecasting if it describes an incoming anomaly not yet reflected in consumption. It may not help if the temperature series and demand history already capture the relevant relationship.

A clinical note may help patient monitoring if it contains symptoms, interventions, or physician observations not yet visible in vitals. It may not help if it merely paraphrases the measurements.

The forecast does not care whether the text is interesting. It cares whether the text changes the prediction.

Real text quality is the bottleneck, not just model quality

The paper extends the complementarity idea with real-world text variants. It creates text based on future information with different levels of explicitness: exact one-step changes, three-step and ten-step average changes, LLM-generated descriptions of future time series, LLM-generated plausible event explanations, and descriptions of historical time series.

The pattern is intuitive and useful. Forecasting performance degrades as the text becomes less directly informative. Historical descriptions, which contain no new information beyond the time-series input, produce the weakest results among those variants.

This is important because many business text sources are not clean predictive signals. They are partial, delayed, political, templated, optimistic, manually entered, or written for humans rather than models. Some are post-hoc explanations. Some are compliance artefacts. Some are copy-pasted theatre. Naturally, all of this is then called “unstructured data”, because “messy institutional residue” does not fit nicely into a slide title.

Before investing in multimodal forecasting, teams should audit text quality along four dimensions:

The model can only align signals that exist. It cannot extract future relevance from a paragraph whose main contribution is confidence.

Prompting is a useful interface, not the default forecasting engine

The paper gives prompting-based methods a serious test. It benchmarks multiple LLMs, optimises prompts, examines in-context learning and chain-of-thought, tests reasoning-oriented models, and even reports fine-tuning results for Mistral on a time-series reasoning corpus.

The result is not that prompting is worthless. Prompt optimisation improves some domains. In-context learning and chain-of-thought sometimes help. Fine-tuning reduces error compared with direct prompting.

But the gap remains. The fine-tuned Mistral model reaches a MASE of 1.702, while Chronos reaches 1.492 in the reported comparison. Bigger or reasoning-tuned language models still struggle to match specialised time-series models.

That should guide deployment design.

Use LLMs to help analysts inspect context, generate hypotheses, summarise drivers, explain anomalies, or prototype forecasting scenarios. Use them cautiously for direct numerical forecasting. When forecasts drive replenishment, staffing, grid operations, financial exposure, or clinical resource allocation, an LLM prompt should not be the only adult in the room.

A good architecture may still include an LLM. It may just not ask the LLM to be the forecaster.

A practical multimodal forecasting playbook

The paper implies a disciplined workflow for businesses considering text-plus-time-series forecasting.

First, beat the unimodal baseline. Start with a strong time-series-only model, not a convenient weak one. If multimodality cannot improve on that baseline, stop celebrating the architecture diagram.

Second, classify the text source. Is the text static or dynamic? Does it arrive before the forecast horizon? Does it map cleanly to the entity being forecast? Does it contain drivers, events, constraints, or only descriptions?

Third, run a redundancy test. Train or evaluate variants where the text is removed, shuffled, truncated, or replaced with historical summaries. If performance barely changes, the text probably is not carrying predictive value.

Fourth, prefer alignment for production accuracy. If enough paired data exists, use alignment-based methods that preserve the time-series backbone. Treat fusion location, projection, pooling, and fine-tuning as model design choices, not implementation trivia.

Fifth, reserve direct prompting for specific roles. Prompting is useful for low-data exploration, analyst copilots, scenario drafting, and qualitative context synthesis. It is less convincing as the primary mechanism for high-stakes numeric forecasting.

Sixth, monitor text drift separately. A time series can drift, and text can drift too. Product descriptions change style. Clinical notes change templates. Market commentary changes vocabulary. Maintenance logs change operators. A multimodal model can degrade because the language channel changes, even if the numeric history looks stable.

The core governance question is not “Do we have text?” It is “Can we prove this text improves the forecast under realistic operating conditions?”

Boundaries: what this paper does not settle

The paper is rigorous, but its scope is specific.

It studies text and time series, not all possible modality combinations. Retail forecasting may involve product images. Industrial forecasting may involve sensor metadata, diagrams, inspection photos, or acoustic signals. Those settings may behave differently.

The benchmark is broad but not exhaustive. Sixteen datasets across seven domains provide serious evidence, not a complete map of every enterprise forecasting problem.

The synthetic tests are mechanism tests, not realistic deployments. They are valuable precisely because they isolate complementarity, but real-world text is noisier, delayed, and more politically contaminated.

The model families are also time-bound. Future forecasting backbones, text encoders, multimodal architectures, and numerically stronger LLMs may shift the frontier. The operating lesson should survive longer than the rankings: benchmark against strong unimodal models, test complementarity, and refuse to pay for text that only repeats the obvious.

The real lesson is not anti-multimodal. It is anti-laziness.

This paper is not a funeral for multimodal forecasting. It is a useful demolition of the lazy version.

The lazy version says: we have numbers, we have text, therefore combining them must be better.

The disciplined version says: we have a forecast target, a time horizon, a numerical history, and a candidate text source. We will test whether the text contains unique predictive information, whether the architecture can use it without damaging temporal modelling, whether the gain survives strong baselines, and whether the added operational cost is justified.

That is less magical. It is also how forecasting systems survive contact with budgets, audits, and Mondays.

Text can help when it tells the model something the numbers do not yet know. When it does not, the best forecasting model may be the one that politely ignores the prose.

Cognaptus: Automate the Present, Incubate the Future.