Tomorrow's Answers, Yesterday's Evidence

TL;DR for operators

An intelligence agent that forecasts tomorrow’s events can learn from today’s completed predictions, but only if it preserves what was knowable when each forecast was made. Otherwise, its apparent improvement may reflect access to answers revealed after the deadline.

EvoForesight investigates three ways to improve such an agent: updating model parameters, retaining reusable forecasting procedures, and generating new questions around observed weaknesses. In controlled experiments, accumulated skills reduced tool calls by approximately 44% on recurring structured tasks while maintaining numeric accuracy. On heterogeneous tasks, they increased searches but improved forecasts.

For forecasting teams, the decision is how to allocate improvement effort across these mechanisms. The paper provides evidence for targeted experimentation, not indefinite autonomous improvement. An operational system still needs timestamped evidence, independently verified outcomes, skill-expiry controls, and cost accounting that includes the computation spent learning from past work.

Learning from an outcome without borrowing its evidence

Consider an intelligence team that issues probability forecasts every morning. By the next day, some events have resolved. The team can compare its predictions with reality, investigate errors, and revise its forecasting process.

There is a temporal problem. Once an event resolves, articles and search results begin reporting its outcome. An AI agent reviewing yesterday’s forecast today might retrieve information that was unavailable when the prediction was originally made. A training system could then reward the agent for using evidence it could never have accessed in a genuine forecast.

In Evolving with Open Worlds, Chen and colleagues introduce EvoForesight, a system designed to turn resolved real-world forecasts into successive improvements while controlling this source of contamination.¹

Its architecture contains three components:

• Prometheus creates forecasting questions from developing events, including daily material collected from 125 websites. Questions undergo temporal validation and other quality checks.
• Branchus produces forecasts through a search team with scheduler, searcher, and critic roles, followed by a prediction team using cached evidence.
• Athena learns from resolved trajectories by updating prediction-model parameters, extracting reusable skills, and changing how future questions are generated.

Branchus separates time-sensitive searching from prediction. The search team records evidence before resolution; the prediction team subsequently operates on those cached records without new external searches.

This separation makes post-resolution training possible without deliberately retrieving revealed answers. It is an architectural safeguard, not independent proof that every cached source was uncontaminated.

The recursive element is equally specific: each round updates model weights, external procedures, or the task-generating environment, and those updated states feed the next round. The system applies predefined learning mechanisms; it does not demonstrate that an AI can invent its own improvement algorithms indefinitely.

Reusable procedures improved forecasts without changing model weights

A forecasting agent can learn how to approach a problem without changing its underlying language model.

Athena examines completed trajectories, evaluates actions, extracts useful procedures, and generalizes them into four levels of reusable skills. At inference time, Branchus retrieves applicable skills, selecting specific guidance where possible and broader guidance for unfamiliar tasks.

The fixed-weight experiments isolate this mechanism from parameter fine-tuning.

Across three nested trajectory pools containing 88, 186, and 262 examples, the reported Brier score fell from 0.135 to 0.122, while numeric mean absolute percentage error (MAPE) declined from 2.80% to 2.28% (Section 2.4, Figure 6). Lower values indicate better forecasts under these measures.

The more revealing result concerns how skill reuse changes searching behavior.

Source: Section 2.5, Figure 8(a). These are task-structure comparisons, not the paper’s main cross-harness benchmark.

For recurring questions, a learned procedure can eliminate redundant investigation. For unfamiliar questions, the same skill mechanism can encourage broader evidence collection.

Fewer calls therefore cannot serve as a universal measure of agent improvement. What matters is whether the additional information changes forecast quality enough to justify its cost.

Cognaptus inference: Enterprise teams operating repetitive intelligence workflows could test skill libraries before committing to frequent model fine-tuning. For less structured research, the pilot should allow additional search when it measurably improves predictions. Neither result establishes that the same behavior will transfer unchanged to another business domain.

Parameter updates improve reasoning over archived evidence

EvoForesight also updates the prediction model itself.

The paper distinguishes reinforcement fine-tuning (RFT), which trains on successful cached trajectories, from reinforcement learning with verifiable rewards (RLVR), which uses resolved outcomes and Group Relative Policy Optimization.

The distinction matters because the RLVR reward checks whether the forecast’s highest-probability outcome was correct. It does not directly reward the quality of the entire probability distribution using Brier score or weighted interval score.

In the parameter-evolution experiments, RLVR increased choice accuracy by 1.3 percentage points for each of two evaluated models. RFT separately produced choice-accuracy improvements ranging from 0.7 to 2.9 percentage points across two models (Section 2.3, Figure 5).

A multi-round RFT experiment on GLM-4.7-Flash reduced Brier from 0.651 to 0.636 and then 0.630 as training examples increased from 600 to 1,000.

These are controlled mechanism results, not additional gains that can simply be added to the skill-evolution improvements. They involve different experimental settings.

For operators, the distinction establishes two independently testable investment choices: improving how an agent gathers evidence through external skills, and improving how its model interprets fixed evidence through parameter updates.

Harder questions can make an unchanged agent score worse

The third improvement mechanism changes the questions the agent encounters.

Athena builds category-level error profiles from resolved forecasts and directs Prometheus toward categories with elevated observed failures. The resulting questions are intended to expose weaknesses that ordinary task generation might underrepresent.

In a fixed-checkpoint comparison, the evolved question environment reduced choice accuracy from 0.60 to 0.53 and increased Brier from 0.52 to 0.58. Confidence-discrimination AUROC remained approximately unchanged, at 0.678 versus 0.680 (Section 2.6, Figure 9(a)).

Because the forecaster checkpoint did not change, these results indicate a harder question distribution, rather than deterioration in the model.

The subsequent training experiment provides a separate test of usefulness. Compared with plain RFT, RFT using adversarially synthesized hard questions reportedly reduced Brier by 1.6%, improved AUROC by 2.7%, and improved relative weighted interval score and median symmetric absolute percentage error by 1.0% and 4.0%, respectively (Section 2.7, Figure 10).

This suggests that selecting more informative training problems can complement learning from existing examples.

It also introduces an evaluation risk. High category-level error rates can reflect intrinsic event uncertainty rather than a correctable reasoning deficiency. A system that generates its own challenges therefore still needs an independent, stable evaluation set to distinguish improved capability from changes in task difficulty.

The leaderboard favors Branchus, with important metric differences

The paper’s main system comparison evaluates 23 harness–LLM configurations on Prometheus-generated forecasting questions, with shared questions and equivalent external tools.

Branchus achieved the highest Bradley–Terry (BT) rating within each of four base-model groups.

Source: Section 2.2, Table 1. Ranks are across all 23 configurations.

BT ratings summarize relative pairwise forecasting performance, combining choice and numerical prediction comparisons. A 10.7-point BT advantage on Qwen3.8-Max is not a 10.7-percentage-point accuracy improvement.

The advantage also varies substantially across models. On DeepSeek-V4-Pro, Branchus leads its strongest same-model baseline by only 0.5 BT points. Other harnesses achieve higher choice accuracy in some model groups, even though Branchus leads in aggregate BT rating.

The authors additionally report first-place results across three consecutive weekly evaluations on the independently refreshed FutureX benchmark. The fourth reported snapshot did not preserve first place. FutureX and the Prometheus-generated comparison are separate evaluations.

One reporting discrepancy deserves explicit treatment: the paper’s narrative describes all four Branchus configurations as top-three overall, whereas Table 1 reports ranks of 1, 2, 3, and 9. The table is the basis for the comparison above.

Moreover, the main comparison does not report question-level sample sizes or uncertainty intervals. These omissions limit how confidently small ranking differences can be interpreted.

Building an improvement loop requires a full cost and audit trail

The architecture suggests a deployment sequence for organizations operating forecast-based intelligence agents.

First, preserve each question, its deadline, the evidence available before that deadline, the forecast probability distribution, and the final outcome. Outcome verification should be independent of the model being improved.

Second, compare procedural skill updates with parameter training on matched forecasting tasks. Track forecast error, search activity, latency, skill freshness, and the computational cost of each learning cycle.

Third, use weakness-targeted questions for development while retaining independently generated evaluation questions. Otherwise, a rising score may reflect changing question selection rather than durable improvement.

The inference-only skill-distillation pipeline is particularly relevant to resource planning. It averages 14.96 model calls and 303,135 tokens per trajectory, including 284,769 input tokens and 18,366 output tokens, with reasoning tokens included in output counts (Section 2.5, Table 2).

Skill extraction avoids gradient updates, but it is not computationally free. Its apparent savings must be measured against extraction, retrieval, verification, and future inference costs.

For investment and financial intelligence teams, the same architecture could be tested on time-stamped, objectively resolvable forecasts. However, better Brier or MAPE scores do not establish profitable trading decisions. That would require independent out-of-sample testing, transaction costs, risk controls, and economic utility measures.

The improvement horizon remains unproven

EvoForesight demonstrates several short-run improvements under controlled computational comparisons. It does not establish that those improvements will continue indefinitely.

The feedback supply is constrained by how quickly events resolve, how reliably outcomes can be verified, and whether newly generated questions remain informative. Skills may also become obsolete as circumstances change. A mistaken outcome label or a contaminated evidence archive could turn the learning process toward the wrong behavior.

There is no complete factorial experiment isolating every interaction among parameter, skill, and environment evolution. Nor does the evidence establish general transfer to software engineering, scientific discovery, or other proposed applications.

The strongest business conclusion is therefore narrower than autonomous self-improvement as a product promise.

EvoForesight makes continuous improvement a system-design problem with three distinct interventions that can be measured separately. Organizations can use that design to test whether accumulated experience improves future decisions, under preserved evidence conditions and a controlled evaluation standard.

Whether the improvement persists—and whether it is worth its full cost—remains a question for deployment evidence.

Cognaptus: Automate the Present, Incubate the Future.