Bellman planning versus learning from experience
Deterministic, offline six-state world. Planning knows the transition/reward model; Q-learning sees sampled transitions only. The left terminal pays 0.4 after two steps from the start; the right terminal pays 1.0 after three.
Experiment 1 — Discounting and Bellman planning
Each move pays the step cost; the final move also pays the terminal bonus. After entering a terminal, there is no future reward. Predict which direction will be preferred before changing the discount factor.
Experiment 2 — Learn Q-values from experience
The learner starts with Q(s,a)=0, acts ε-greedily, observes one transition and reward at a time, and updates only the visited state-action pair. The Bellman benchmark is shown for comparison—not used in training.
Displayed rewards are discounted *episode returns during training*, not unbiased off-policy estimates. Each run restarts with a fixed seed and identical zero Q-values. Alpha is fixed at 0.4, and each episode has a 30-step safety cap.
Experiment 3 — Policy-dependent data and missing support
Compare ε=0 and ε=0.25 at the selected discount and step cost, each with the same number of episodes. A tie is broken toward Left; zero-initialized values can still induce isolated trials of previously untested actions.
Higher visitation coverage is evidence about observed actions, not proof of correct off-policy evaluation, model identification, or superior deployment outcomes.