Stuck on Repeat: When Reinforcement Learning Fails to Notice the Rules Changed

A dashboard still looks the same after the business changes.

The buttons are in the same place. The form fields have the same labels. The workflow still asks for the same approval, the same handoff, the same final action. From the outside, nothing has moved.

Then the rules underneath change.

A supplier starts behaving differently after a policy shift. A trading market reacts differently after a liquidity regime changes. A robot arm keeps seeing the same objects, but the hardware has worn slightly. A customer-service automation still receives the same message types, but the escalation logic behind the organization has quietly changed.

This is where many reinforcement learning systems look less like adaptive intelligence and more like expensive muscle memory.

The paper behind TAPE asks a clean version of this uncomfortable question: what happens when an RL agent faces the same interface, the same action space, the same goal, and the same reward structure, but the hidden transition rule changes?¹ No visual distractions. No new task labels. No reward reshaping drama. Just a different law of motion under the same surface.

That makes the paper useful. Not because business teams need cellular automata in production — thankfully, most procurement departments are not binary tapes evolving under Rule 30 — but because TAPE isolates a failure mode that is easy to hide in messier benchmarks: the agent may learn how to act inside familiar rules without learning how to infer the rules.

And that distinction is the whole story.

The interface is fixed; the physics is not

TAPE is built from one-dimensional elementary cellular automata. The environment contains a binary tape. At each step, the agent can flip one cell. After the agent acts, the tape evolves according to a cellular automaton rule. The goal is to reach a target tape, such as the all-zero tape.

The important part is not the tape. It is the experimental surgery.

The authors hold the visible interface fixed:

During training, episodes are sampled from one set of rules. During out-of-distribution evaluation, the agent faces disjoint held-out rules. The agent is not being asked to generalize to a different-looking world. It is being asked to generalize to a different-working world.

That is a sharper diagnostic than the usual “OOD” label, which often bundles together changes in visuals, dynamics, reward functions, goals, and task composition. When such an agent fails, everyone gets to argue about what actually caused the failure. Very academic. Very convenient.

TAPE removes much of that ambiguity. If performance degrades, the main suspect is the latent rule shift.

This is why the benchmark’s minimalism is not a weakness by default. A simple environment can be a bad model of the world, yes. But it can also be a good microscope. TAPE is closer to the second case. It does not claim to simulate business reality. It creates a small chamber where one mechanism can be observed without five other mechanisms photobombing the result.

The agent has two jobs, but most benchmarks only test one

In TAPE, a good policy needs more than action optimization.

It needs to do two things:

1. Choose actions that move the tape toward the goal.
2. Infer, implicitly or explicitly, which hidden rule is currently governing the tape.

Those are not the same skill.

An agent can learn that certain interventions tend to help under the training rules. It can memorize useful regularities, fit local correlations, and produce decent in-distribution behavior. But when the rule changes, those correlations may stop being useful. The same action can now have a different downstream consequence.

That is the mechanism-first reading of the paper:

Same interface
      ↓
Different latent transition rule
      ↓
Old action regularities become unreliable
      ↓
Policy quality now depends on latent-rule inference

This matters because many deployed AI systems are evaluated as if the first job were enough. We test whether the agent makes good decisions on historical cases, simulated environments, or familiar operational regimes. Then we quietly assume that “good decision-making” will transfer when the causal machinery changes.

TAPE says: not so fast.

The system may not have learned a transferable decision principle. It may have learned a response pattern conditioned on the old dynamics. That is fine until the world updates without changing the user interface. Which, annoyingly, is one of the world’s favorite hobbies.

Low success rates are not automatically a benchmark failure

A lazy reading of the results would say: the success rates are low, so maybe the benchmark is just too hard or too artificial.

The paper anticipates this objection, and this is one of its more useful contributions. It does not simply report poor RL performance and declare victory. It tries to separate three different explanations:

The main evaluation uses a budgeted true-dynamics random-shooting MPC reference. This planner knows the transition rule and samples candidate action sequences, but it is still finite-budget. It is not a mathematical proof of optimal solvability. It is a protocol-matched operational reference.

Under the default protocol, this true-dynamics MPC reference reaches 18.7% strict success. The paper also reports a smaller feasibility regime where rule-wise solvability reaches 100%, helping show that the environment family is not simply impossible by construction.

That distinction is important. In sparse-control settings, strict success can be harsh. Exact goal-reaching may be rare even when the agent improves the state. TAPE therefore reports not only strict success but also continuous metrics: final distance, trajectory-level AUC distance, and soft success at a tolerance threshold.

This is good evaluation hygiene. Sparse success rates are informative, but they are also blunt. Continuous distance metrics show whether the agent is getting closer even when it does not land exactly on the goal. The paper’s point is not “everything failed, therefore benchmark good.” The point is that performance should be read through calibrated references, not just raw percentages floating in space looking scientific.

The main result: every baseline loses ground under held-out rules

The reported baseline suite covers several families:

The headline pattern is consistent: all reported methods suffer an ID-to-OOD drop under held-out rule evaluation.

In the main 20-seed comparison, DQN drops from 0.073 ID success to 0.048 OOD success. DQN+CURL drops from 0.080 to 0.056. DQN+RAD drops from 0.082 to 0.056. PPO drops from 0.070 to 0.042. DreamerV3 drops from 0.062 to 0.036. PEARL-DQN drops from 0.074 to 0.048.

These are small absolute numbers, so the percentages should not be overdramatized. The more important point is the pattern: the drop appears across method families, with bootstrap confidence intervals reported over 20 seeds. The paper treats uncertainty as part of the result, not as an appendix-shaped confession.

A compact version of the main comparison looks like this:

The continuous metrics tell the same broad story. Final distance and AUC distance worsen from ID to OOD, while soft success decreases. This matters because it reduces the risk that the strict success metric alone is creating a misleading floor effect.

The result is not that one specific algorithm embarrasses itself. That would be too easy. The result is that multiple families show the same transfer gap when the hidden law changes and the visible interface does not.

Augmentation helps with surfaces; rule inference needs more than surfaces

RAD and CURL perform among the better baselines in the main comparison. That is not surprising. Augmentation and contrastive objectives can help the agent learn more stable representations. If the problem were mainly about nuisance variation in observations, this would be an encouraging sign.

But TAPE is not mainly about nuisance variation.

The hidden rule changes the transition structure. A representation that is invariant to bit shifts or bit flips may be less brittle, but it still may not identify the active rule well enough to plan under that rule. Regularization can make the agent smoother. It does not necessarily make it mechanistically curious.

This is a useful business distinction.

Many robustness techniques used in AI systems are surface stabilizers. They reduce sensitivity to input noise, formatting differences, mild perturbations, or familiar types of variation. That is valuable. But it is not the same as adaptation to a changed operating mechanism.

For example, a workflow agent may become robust to different invoice layouts. Wonderful. Very adult. But if the company changes its approval policy, supplier-risk logic, or exception-routing rules, layout robustness will not save the agent. It needs to infer the new process logic.

TAPE gives this distinction a clean experimental form. Augmentation can improve robustness around the interface. Rule-shift generalization requires understanding how actions cause consequences under a latent law.

Those are adjacent skills, not identical ones.

Task inference helps most clearly when the rule is identifiable

The paper includes supplementary diagnostics that make the task-inference story more nuanced.

In a smaller $L=H=16$ regime, the authors compare multi-rule training, fixed-rule runs, and true-dynamics planning. At this scale, the paper reports that all rules are feasible in principle under the feasibility sweep. The results show that PEARL-style task inference can substantially improve difficult fixed-rule cases compared with RAD.

For representative rules, PEARL reaches 1.00 strict success on rules 108 and 204 where RAD is much lower, and reaches 0.95 and 0.94 on chaotic rules 30 and 110 where RAD is near zero. In multi-rule training at this smaller scale, PEARL also improves over RAD: 0.35 versus 0.28 strict success.

This is not the same as saying PEARL solves TAPE. In the main larger regime, PEARL-DQN still shows an OOD drop and does not dominate. But the smaller diagnostic is valuable because it identifies a plausible mechanism: when the latent rule can be inferred from context and the regime is controllable enough, context-based task inference can help.

That is exactly the kind of result a benchmark should produce. Not just a leaderboard. A mechanism diagnosis.

The practical translation is straightforward: if an agent must operate under changing hidden rules, then “more training data” is not always the right abstraction. The system may need an explicit belief state, memory, posterior update, or rule-identification module. Otherwise, it may keep optimizing actions as if the old causal model still held.

Business people call that “experience.” Engineers call it “distribution shift.” In meetings, both are usually followed by a dashboard nobody trusts.

World models are not automatically safer just because they can imagine

World models have an appealing story: learn the dynamics, simulate futures, choose better actions. In stable environments, this can be powerful. In shifted environments, imagination can become a liability if the learned dynamics are wrong.

TAPE’s DreamerV3-style baseline underperforms several value-based variants in the reported main comparison. The appendix adds credibility diagnostics: one-step prediction error is higher on OOD rules than ID rules, rollout error grows sharply with planning depth, and increasing imagination horizon beyond a certain point produces limited gains under rule shift. The reported OOD strict success for the DreamerV3-style baseline is also far below the 18.7% budgeted true-dynamics MPC reference.

The paper is careful here. It does not prove that all world models must fail under rule shift. It evaluates one representative DreamerV3-style implementation under a fixed benchmark protocol. That boundary matters.

Still, the mechanism is familiar: learned models can compound error over imagined rollouts. If the model’s transition assumptions are slightly wrong, planning deeper can amplify the mistake. In ordinary language, the agent becomes very good at thinking ahead inside the wrong world.

For business systems, this is the difference between planning and trustworthy planning. A planning agent that does not know when its model is stale is not robust. It is just confidently obsolete.

A more useful design direction is model-usage control: estimate when the learned model is reliable, shorten imagination depth when uncertainty rises, route to safer reactive policies when needed, or take actions that improve belief about the current mechanism before committing to high-impact decisions.

That is not as glamorous as “the agent plans.” But it is closer to what operational robustness requires.

The rule types explain why aggregate scores can mislead

TAPE’s cellular automaton rules are grouped into operational types: stable, periodic, and chaotic. Stable rules approach fixed patterns. Periodic rules oscillate. Chaotic rules produce irregular dynamics.

This classification matters because the held-out split is not type-balanced in the main default protocol. The paper reports that the test set is dominated by chaotic rules: 22 of 30 held-out rules in the default farthest-point split. That composition affects aggregate strict-success rates.

The appendix’s by-type MPC diagnostic makes the point even sharper. Under one fast-replication by-type evaluation, true-dynamics MPC reaches 0.50 strict success on stable rules, 0.17 on periodic rules, and 0.00 on chaotic rules, with the aggregate dominated by the chaotic bucket. The headline MPC reference in the main table uses a different fuller evaluation budget, so these numbers should not be mixed casually. The paper explicitly warns against treating protocol-specific references as interchangeable.

This is not a boring technicality. It is one of the paper’s central evaluation lessons.

A single OOD score can hide the fact that some mechanisms are tractable, some are difficult, and some are nearly unreachable under the chosen horizon and planner budget. In business evaluation, the same problem appears when teams report one aggregate automation success rate across easy cases, edge cases, adversarial cases, and rare-but-expensive failures.

Averaging is not analysis. It is often where analysis goes to retire.

TAPE’s per-rule and per-type breakdowns are therefore not decorative. They help determine whether the agent is failing everywhere, failing mainly on chaotic dynamics, or failing because the evaluation split overweights hard mechanisms. That is the kind of diagnostic separation most agent benchmarks still need more of.

The appendix tests interpretation, not a second thesis

A strong paper appendix should not be a dumping ground for things the authors could not fit into the main story. Here, the appendix mostly serves a clearer purpose: it protects the interpretation of low success and OOD degradation.

Several appendix components are especially useful:

This matters because the most common misreading of TAPE is: “The success rates are low, therefore the benchmark is just punishing.”

A better reading is: the authors use oracle calibration, smaller feasibility regimes, fixed-rule runs, continuous metrics, and by-type analysis to identify what kind of difficulty they are measuring. Some difficulty comes from reachability. Some comes from chaotic dynamics. Some comes from finite planning budget. And some comes from learned agents failing to transfer across latent rules.

That separation is the contribution.

Without it, TAPE would indeed be just another hard benchmark with a clever name. With it, the benchmark becomes a diagnostic instrument.

What Cognaptus infers for business AI evaluation

The paper directly shows that, in a controlled cellular automata RL benchmark, several representative RL families degrade under held-out latent transition rules, even when observations, actions, goals, and rewards are fixed.

Cognaptus infers a broader evaluation principle:

When an AI agent operates in a changing business environment, testing should distinguish interface familiarity from mechanism familiarity.

That principle applies beyond reinforcement learning labs. Consider three business contexts.

First, workflow automation. An agent trained on historical process traces may learn how approvals usually move through an organization. But if compliance rules, risk thresholds, or staffing authority changes, the interface may look identical while the transition logic changes. Testing only on old traces rewards procedural imitation, not rule adaptation.

Second, trading and investment automation. A strategy may perform well under one liquidity regime, volatility structure, or market microstructure. If the hidden regime changes, the same observed price-action patterns can imply different future consequences. The agent needs regime inference, not just signal memorization with nicer charts.

Third, robotics and physical automation. A robot may observe the same workspace and use the same action primitives, but friction, wear, calibration drift, or object properties can alter dynamics. A policy that treats those shifts as noise may fail in exactly the cases where adaptation matters.

The practical implication is not “use TAPE before deploying invoice bots.” Please do not make the procurement department suffer unnecessarily.

The practical implication is to borrow TAPE’s evaluation pattern:

That is the business value of the paper. Cheaper diagnosis. Better stress testing. Fewer agents that look competent because the test environment politely refuses to surprise them.

Where the result should not be overextended

TAPE is deliberately narrow. It uses one-dimensional elementary cellular automata, deterministic transitions, binary tapes, and controlled rule splits. This makes attribution clean but ecological coverage limited. Real business environments contain noisy observations, partial observability, multi-agent feedback, changing incentives, delayed rewards, and human behavior — a charming swamp, as usual.

The evaluated baseline set is also not exhaustive. The paper includes model-free, augmentation-based, task-inference, and one DreamerV3-style world-model family, but it does not close the case on recurrent controllers, explicit Bayesian POMDP solvers, richer world models, offline sequence decision models, or more specialized automata-aware architectures.

The benchmark’s compute discipline is another boundary. Twenty-seed evaluation, split sweeps, horizon tests, and per-rule diagnostics improve inferential stability, but they raise the cost of participation. The paper notes the need for reduced-budget tracks that preserve comparability.

Finally, the oracle-normalized score is protocol-specific. A value calibrated against one rule mix, horizon, initialization distribution, and planner budget should not be treated as universal. If the protocol changes, the reference must be re-estimated. Apparently, even oracles need paperwork.

These limitations do not weaken the core lesson. They define it. TAPE is not a universal simulation of intelligence. It is a controlled test for latent-rule generalization.

The uncomfortable lesson: action skill is not rule understanding

The easiest mistake is to treat reinforcement learning success as evidence that the agent understands the environment.

TAPE pushes back. An agent may optimize behavior under familiar transition laws and still fail when the latent law changes. It may see the same tape, use the same action, receive the same reward, and pursue the same goal — while quietly acting inside the wrong model of the world.

For AI teams, that is the warning worth keeping.

Do not only ask whether the agent performs well when the world looks familiar. Ask whether it notices when the world works differently.

Because an agent that cannot detect changed rules is not robust. It is just stuck on repeat, waiting for reality to stop being inconvenient.

Cognaptus: Automate the Present, Incubate the Future.