Stacking Alpha: How HARLF's Three-Tier Reinforcement Learner Beats the Market

TL;DR for operators

HARLF is not a story about a large language model suddenly becoming a portfolio manager. Sensible readers may exhale. The language component is FinBERT sentiment scoring applied to financial news, then converted into monthly asset-level signals. The heavier claim is architectural: instead of throwing price metrics and sentiment into one flat reinforcement-learning model and hoping the neural soup tastes like alpha, the paper separates the decision process into three tiers.

At the bottom, base reinforcement-learning agents propose allocations from either structured market metrics or NLP-derived sentiment observations. In the middle, meta-agents aggregate the recommendations within each modality. At the top, a super-agent combines the market-metric view and the sentiment view into final portfolio weights. That is the paper’s useful idea: not “LLM beats market,” but “separate noisy modalities, let specialised agents form opinions, then learn how to arbitrate between those opinions.”¹

The headline result is strong. On a 2018-2024 backtest across 14 equity and commodity assets, the super-agent reports 26.0% annualised ROI, a 1.2 Sharpe ratio, and 20.0% annualised volatility. Equal weights report 7.5% ROI and 0.57 Sharpe; the S&P 500 benchmark reports 13.2% ROI and 0.63 Sharpe. The NLP meta-agent also does well: 20.5% ROI, 1.2 Sharpe, and 16.0% volatility, compared with the metrics-only meta-agent’s 14.7% ROI and 0.8 Sharpe.

For operators, the immediate lesson is not to deploy this as a robo-CIO before lunch. The practical lesson is to design financial AI systems with modality boundaries, audit paths, and staged aggregation. HARLF gives a useful research blueprint for combining textual sentiment with numerical market indicators. It does not yet answer the questions that decide whether a strategy survives contact with brokers, fees, slippage, delayed news, stale sentiment, and the occasional market tantrum pretending to be a structural break.

The useful trick is not “AI trading”; it is staged disagreement

Portfolio AI has a familiar failure mode: collect many signals, concatenate them into a heroic feature vector, train a model, and then act surprised when the system becomes difficult to interpret, unstable across regimes, or quietly overfit. Finance loves complexity, especially when complexity comes with a Greek letter and a backtest.

HARLF takes a more organised route. It does not ask one model to learn everything from everything at once. It creates disagreement first, then learns how to resolve it.

That difference matters. In investment work, a price-based model and a news-sentiment model are not merely two versions of the same evidence. They answer different questions. Market metrics ask: what has price, risk, correlation, and drawdown already done? Sentiment asks: what tone is the public information environment carrying around the asset? These signals overlap, but they do not arrive with the same timing, noise, or bias. Compressing them too early can hide the very contradiction that might matter.

HARLF’s hierarchy keeps that contradiction visible:

This is why the paper is best read mechanism-first. The return table is interesting, but the architecture is the point. The model’s reported outperformance is the evidence; the design pattern is the reusable idea.

HARLF builds two separate senses before giving the system a brain

The paper’s first pipeline is numerical. It collects daily adjusted closing prices for 14 assets from January 2003 to December 2024 using Yahoo Finance through yfinance. From those prices, it computes monthly observation vectors: Sharpe ratio, Sortino ratio, Calmar ratio, maximum drawdown, volatility, and a flattened correlation matrix across assets. This gives the data-driven agents a view of recent risk-return behaviour and cross-asset relationships.

The second pipeline is textual. For each asset and month, the system queries Google News using asset-specific terms, scrapes the first ten articles for each link, extracts article text, and applies FinBERT to classify financial sentiment. The monthly sentiment score is computed from the difference between positive and negative sentiment probabilities, normalised by article count. The NLP observation vector combines this sentiment score with volatility.

This design choice is more conservative than the “LLM-driven” label may suggest. HARLF does not ask a general-purpose model to reason through filings, generate a macro thesis, or explain a central-bank press conference while wearing a tiny portfolio-manager hat. It uses a domain-specific transformer as a sentiment classifier, then feeds the resulting signal into reinforcement learning.

That is not a criticism. In production finance, smaller and specialised often beats larger and theatrical. FinBERT can be cheaper, faster, and easier to govern than a general-purpose generative model. The awkward phrase “lightweight LLM” is doing real work here: the language model is a signal extractor, not the final decision-maker.

The paper then trains reinforcement-learning agents to output portfolio weights under long-only, no-leverage constraints. The action space is continuous, so agents can allocate different weights across assets rather than choosing from crude buy/sell/hold categories. The reward function is a linear combination:

where ROI is return on investment, MDD is maximum drawdown, and $\sigma$ is volatility. The coefficients vary between 0.5 and 2 in the reported experiments, with maximum drawdown heavily penalised. That reward choice matters: the model is not simply maximising return. It is being taught that ugly paths to return are expensive. A rare outbreak of financial common sense.

The hierarchy turns many unstable opinions into fewer auditable decisions

At the base layer, HARLF uses four reinforcement-learning algorithms from Stable Baselines 3: PPO, SAC, DDPG, and TD3. These are standard continuous-control algorithms, each with a different learning style. PPO is usually valued for stable policy updates. SAC encourages exploration. DDPG and TD3 are actor-critic methods for continuous actions, with TD3 designed to reduce overestimation problems that can affect DDPG.

The paper runs these base agents across both the metrics-based and NLP-based settings. It also uses fixed consecutive seeds and reports median base-agent results across five seeds. That seed detail is not decorative. Reinforcement learning can be temperamental; multiple seeds are a minimum defence against mistaking one lucky training run for intelligence.

The middle layer is where HARLF becomes more than a collection of agents. A meta-agent receives the action vectors proposed by base agents and learns to output a final allocation for that modality. Instead of averaging the base agents, it learns which recommendations deserve more weight in different contexts.

That distinction is important. A weighted average assumes the aggregation rule is mostly static. A learned meta-agent says: perhaps PPO’s allocation is useful in one environment, TD3’s in another, sentiment in one regime, risk metrics in another. The meta-agent is a small neural network with ReLU activations and a softmax output layer, so the final weights remain non-negative and sum to one.

Then the super-agent repeats the same logic at the top. It receives the allocation proposed by the metrics meta-agent and the allocation proposed by the NLP meta-agent. Its job is to learn the final arbitration rule between market state and sentiment state.

This is the core mechanism. HARLF is not asking sentiment to replace market data. It is asking sentiment to compete with market data in a structured way. That is much closer to how investment organisations actually operate. Quant desks, discretionary analysts, macro researchers, and risk teams often disagree. The business problem is not eliminating disagreement. The business problem is deciding which disagreement is useful.

The evidence supports the architecture, but not the fairy tale

The paper’s main evidence is the 2018-2024 backtest. The agents are trained on the earlier historical period described in the methodology as 2003-2017, then tested on later data. The portfolio contains 14 assets: major equity indices across the US, Europe, Hong Kong, China, India, South Korea, plus gold, silver, and WTI crude oil futures.

The results are clean enough to deserve attention:

Three observations matter more than the obvious “26% is bigger than 13.2%.”

First, the NLP meta-agent beats the metrics meta-agent. In the reported test period, the sentiment-driven branch achieves 20.5% annualised ROI and 1.2 Sharpe with 16.0% volatility. That is materially stronger than the metrics meta-agent’s 14.7% ROI and 0.8 Sharpe. If replicated, this would suggest that monthly news sentiment is not just noise when integrated through a disciplined RL process.

Second, the super-agent improves ROI but does not improve Sharpe beyond the NLP meta-agent. The super-agent reaches 26.0% ROI, but its Sharpe remains 1.2 while volatility rises to 20.0%. That means the top layer appears to extract more return while accepting more volatility. This is not automatically bad; a CIO may take that trade. But it is not a free lunch. Even artificial intelligence, despite its branding department, does not repeal arithmetic.

Third, the metrics-only meta-agent barely beats the S&P 500 on ROI but does so with lower volatility and a higher Sharpe. That matters because it separates absolute return from risk-adjusted quality. In portfolio terms, the paper is not merely showing “more return”; it is showing different return-volatility trade-offs across the hierarchy.

The granular results also help interpret the role of aggregation. Among metrics base agents, TD3 performs best at 16.5% ROI and 0.8 Sharpe, while the metrics meta-agent reports 14.7% ROI and 0.8 Sharpe. Among NLP base agents, TD3 again leads the base group at 17.5% ROI, but the NLP meta-agent improves to 20.5% ROI and 1.2 Sharpe. The aggregation layer appears more useful in the NLP branch than in the metrics branch.

That is not a universal law. It is a reported result in this paper’s dataset. Still, it is suggestive: noisy sentiment signals may benefit more from learned aggregation than cleaner numerical metrics do. A good article could stop there. Fortunately, we have standards.

The comparison table is context, not a coronation

The paper also compares HARLF with prior RL portfolio strategies. This is best treated as contextual comparison, not a definitive leaderboard. Different studies use different assets, periods, costs, constraints, data handling, and backtesting assumptions. Comparing annualised ROI across papers is useful for orientation, but it is not an Olympic final.

The paper reports the following comparison:

The interesting comparison is not that HARLF “wins” on ROI. It matches the DQN result on ROI while reporting much lower volatility, and it exceeds the CNN-RL ROI while slightly trailing its Sharpe. That suggests HARLF is competitive within the reported comparison set. It does not prove dominance over the state of the art in the broad market, because broad market dominance requires a common benchmark protocol. Finance papers do enjoy making apples race oranges and calling the winner a pear.

The correct interpretation is narrower and stronger: HARLF’s architecture produces performance in the same range as recognised RL portfolio methods, while adding a modular sentiment channel and open reproducibility notebooks. For research and prototyping, that is useful.

What the experiments are actually doing

The paper contains several pieces of evidence that serve different purposes. Mixing them together makes the work look either stronger or weaker than it is.

This distinction matters for operators. A main result can justify attention. A comparison table can justify further benchmarking. Open notebooks can justify faster experimentation. None of those automatically justify capital allocation.

The business value is signal governance before trade automation

The obvious business pitch is “AI portfolio system beats benchmarks.” The useful business pitch is quieter: HARLF offers a pattern for governing heterogeneous signals.

A financial institution, wealth platform, or quant research team could use this architecture to prototype multimodal allocation systems without forcing every signal into one opaque model. The base agents preserve specialist views. The meta-agents create modality-level outputs. The super-agent provides a controlled arbitration layer. Each layer can be inspected, compared, replaced, or stress-tested.

That has several operational consequences:

For an operator, the most valuable question is not “Can HARLF be traded tomorrow?” It is “Can this architecture help us separate signal creation, signal aggregation, and final allocation in a way our risk committee can understand?” That is a better question because it survives the first meeting with compliance.

A practical deployment path would begin as decision support. Run the system in shadow mode. Compare its allocations with current portfolios. Track when the sentiment meta-agent disagrees with the metrics meta-agent. Ask whether those disagreements anticipate drawdowns, rotations, reversals, or simply media noise. Only after that should anyone discuss capital, and even then with transaction costs and kill switches included. Revolutionary, I know: test the thing before letting it touch money.

The misconception: this is not proof that LLMs beat markets

The title of the paper and the headline numbers tempt a familiar interpretation: language models are now market-beating portfolio managers. That interpretation is too blunt.

First, the LLM component is FinBERT sentiment classification, not generative reasoning. The model turns news articles into monthly sentiment scores. It does not produce a narrative thesis, evaluate uncertainty, detect source credibility, or infer causal structure across macro events.

Second, the test is historical. The system is trained and evaluated on market history, with the main backtest covering 2018-2024. Backtests are necessary, but they are not reality. They are reality with the operational inconvenience removed.

Third, the system excludes transaction costs. Monthly rebalancing reduces turnover concerns compared with daily trading, but it does not erase them. A strategy that reallocates across futures, indices, and commodities still faces spreads, fees, tax effects, implementation vehicles, and liquidity constraints.

Fourth, the text pipeline assumes sentiment data is available in a clean monthly form. In live markets, news arrives asynchronously. Sometimes news moves after price. Sometimes price moves before journalists find adjectives. Sometimes the most important information is not in the article text but in the timing, source, revision, or market expectation. The paper recognises this boundary and lists asynchronous integration as future work.

So the replacement belief should be more precise: HARLF is evidence that a lightweight financial NLP signal can be integrated into a hierarchical RL allocation framework and produce strong historical backtest results under the paper’s assumptions. That is useful. It is not a licence to rename the investment committee “the prompt engineering department.”

Where the result is strongest, and where it thins out

HARLF is strongest as a system-design paper. It gives a concrete architecture, a defined asset universe, a reproducible pipeline, and a clear result table. The hierarchy is intuitively aligned with how investment decisions are already organised: specialist inputs, modality aggregation, final allocation.

It is also strongest in showing that NLP sentiment should not be dismissed as a decorative feature. In the reported test, the NLP meta-agent is not a minor add-on. It is one of the main performers. The super-agent’s extra ROI comes from combining that sentiment branch with the metrics branch.

The evidence thins out around real-world implementation. The paper does not model transaction costs. It does not test asynchronous news arrival. It does not provide adversarial stress tests or extreme-event simulations. It does not show live paper-trading results. It does not deeply analyse turnover, attribution, regime sensitivity, or whether returns are concentrated in a small number of assets or periods.

Those omissions do not invalidate the paper. They define the next validation checklist.

The correct business posture is therefore disciplined curiosity. HARLF deserves replication, stress testing, and adaptation. It does not deserve blind deployment.

The sharper lesson for financial AI builders

The broader lesson is that financial AI systems should not be judged only by model class. “Uses RL” is not enough. “Uses LLMs” is definitely not enough. The architecture around the models often determines whether the system is inspectable, maintainable, and capable of controlled disagreement.

HARLF’s hierarchy gives builders three useful principles.

First, preserve modality boundaries long enough to learn their value. If sentiment and price metrics are merged too early, you may lose the ability to diagnose which signal is doing the work. That is bad research and worse governance.

Second, aggregate decisions, not just features. HARLF’s meta-agents consume proposed allocations from base agents. This means the system learns from agent behaviour, not merely raw inputs. That creates a modular path: swap a base agent, add a data source, test a new sentiment model, or adjust the final arbitration layer.

Third, align the action space with portfolio constraints. Long-only, no-leverage, softmax-normalised weights make the experiment easier to interpret. Constraints are not a boring appendix. They are where trading fantasies become portfolios.

The irony is that the “LLM” part of the paper is less futuristic than the organisational design. FinBERT sentiment scoring is useful, but the hierarchy is the idea worth stealing. Carefully, legally, and preferably after risk review.

Conclusion: alpha may be stacked, but validation cannot be skipped

HARLF’s reported results are impressive: 26.0% annualised ROI and 1.2 Sharpe for the super-agent over 2018-2024, with the NLP meta-agent also showing strong risk-adjusted performance. The paper’s more durable contribution is its architecture: a staged reinforcement-learning system that separates market metrics from news sentiment, aggregates each modality, and then learns how to arbitrate between them.

That architecture is business-relevant because it maps onto a real operating problem. Financial firms do not merely need more signals. They need ways to compare signals, govern signals, and understand when one signal should overrule another. HARLF gives a credible prototype for that kind of system.

But the result should be read with boundaries intact. The backtest excludes transaction costs, assumes synchronised data, and does not establish live robustness. Sentiment can add signal, but it can also add media bias with a charming user interface. The super-agent improves return, but with higher volatility than the meta-agents. The paper is promising; it is not supernatural.

The practical next step is not a production trading bot. It is a replication and shadow-testing programme: rerun the notebooks, add costs, test turnover, stress the news pipeline, analyse attribution, and watch how the super-agent behaves when sentiment and market metrics disagree. That is where HARLF’s real value will be decided.

Cognaptus: Automate the Present, Incubate the Future.