Replan, Rethink, Repeat: Why Vision-Language Models Make Better Closed‑Loop Planners

Opening — Why this matters now

Robotics is rediscovering an old truth: it’s not the plan that matters, it’s the replanning.

As more companies experiment with Vision-Language Model (VLM)-driven robotic agents—from warehouse pickers to home-assistance prototypes—a quiet tension is emerging. These models can generate impressively detailed symbolic plans, but their reasoning occasionally drifts into the surreal. You can’t ship a robot that confidently places lemons after oranges simply because the model had an off day.

This paper tackles an under‑examined but deeply practical question: How should we structure VLM‑based planners so they make fewer mistakes over long horizons? The researchers take a page from Model Predictive Control (MPC), treating LLM/VLM reasoning as a noisy, imperfect controller that needs regular correction.

Spoiler: closed‑loop beats open‑loop, warm‑starting beats cold‑starting, and shorter control horizons aren’t always the blessing you think they are.

Background — Context and prior art

The idea of using LLMs and VLMs for symbolic planning has been around since early PaLM‑E and SayCan experiments. These systems relied on large pre‑trained models to reason about objects, tasks, and simple constraints in tabletop manipulation.

But nearly all of these works rely, implicitly or explicitly, on closed‑loop planning: generate a plan, execute one or two steps, observe, re‑ask the model.

What has been missing is: a principled, control‑theoretic evaluation of how often to replan, whether to warm‑start, and what failure patterns emerge.

This paper fills that gap across four environments of escalating complexity—from colored cubes to fruit salad with quirky constraints—and three competitive VLMs.

Analysis — What the paper actually investigates

The authors structure the investigation around three core questions:

1. Closed‑loop vs. open‑loop: Is there measurable value in letting the VLM replan instead of committing to the first plan it produces?
2. Control horizon: How frequently should the planner replan—after every action? Every few steps? Only on failure?
3. Warm‑starting: Should we feed the model its previous plan and partial execution trace when asking it to replan, or is it better to start fresh?

To isolate these factors, the authors:

• Test 3 VLMs across 4 manipulation tasks
• Run 50 randomized trials per scenario
• Measure geometric success, logical consistency, and full task completion
• Use two‑proportion z‑tests to establish significance (rare in LLM robotics papers—refreshing)

The key move is to treat VLM planners as stochastic closed‑loop controllers, subject to noisy inference. This framing allows systematic comparison rather than anecdotal performance claims.

Findings — Results that actually matter

The punchlines are clear and surprisingly pragmatic.

1. Closed‑loop planning beats open‑loop—even in static environments.

Open‑loop planners fail quietly: they generate a perfect-looking plan that collapses the moment an early misplacement cascades forward. Closed‑loop versions improved geometric success by 21.7% on average.

In other words: don’t trust a single-shot plan from a VLM. Ever.

2. Shorter control horizons don’t reliably improve performance.

Conventional wisdom says: replan more often → more robustness.

The data says: not really.

CL-SHORT (replan every ~2 steps) is “best” on paper, but only 2 of 12 scenarios show statistically significant improvement.

Why? Two forces cancel each other out:

• More replanning = more opportunities to fix mistakes
• More replanning = more opportunities for the VLM to introduce new mistakes

A frontier robotist’s version of, “More meetings don’t always solve things.”

3. Warm-starting is dramatically effective.

This was the most striking result.

Providing the previous plan + execution feedback made planners:

• 28.2% better at full task completion
• 31.7% better at geometric reasoning
• Significantly less likely to introduce new logical errors

The few scenarios with no warm‑start collapsed completely, especially in more complex tasks.

Warm‑starting is essentially giving the VLM a memory of its prior intent—reducing drift, adding coherence, and preventing it from “hallucinating a new task mid‑way.”

4. Logical reasoning remains the Achilles heel.

Even with closed‑loop updates and warm‑starting, VLMs often struggle more with logical ordering than with geometry.

Logical correction rates remained inconsistent, and warm‑starting tended to reduce both positive and negative corrections—because the model became anchored to prior plans.

Overall Recommendation from the authors

Use closed‑loop planners with warm‑starting. Choose a control horizon that’s “reactive enough,” not maximal.

A refreshingly non‑dogmatic answer.

Visualization — A quick summary

Table: What Each Strategy Helps (✓) or Hurts (✕)

Warm‑starting is the only unambiguously positive technique.

Implications — Why businesses and robotics teams should care

This study translates directly into practical decision rules for anyone building VLM‑driven automation.

1. Don’t deploy single‑shot LLM plans in physical systems.

If your autonomous agent executes a full plan without reevaluating, you’re betting the entire operation on one noisy forward pass.

2. Replanning frequency is a cost–benefit curve, not a maximization problem.

More replanning increases inference cost and error exposure. There is no need to replan at every micro‑step unless the environment is extremely dynamic.

3. Warm-starting is the cheapest reliability upgrade available.

Warm-starting is trivial to implement but drastically increases task stability. Any agentic system that doesn’t incorporate prior-action context is leaving performance on the table.

4. Logical constraint handling remains a major gap.

Expect future gains from:

• Fine‑tuning on constraint-heavy tasks
• Hybrid methods combining symbolic solvers with VLMs
• Model distillation from classical planners

This paper essentially confirms what many practitioners already suspect: VLMs are good at visual grounding but brittle at rule-following without scaffolding.

Conclusion — Wrap-up

In the same way MPC revolutionized real-world control systems by reintroducing feedback into planning, this work pushes VLM-based robotic planning toward similar maturity. Replanning helps, warm‑starting helps even more, and the fetish for extremely short control horizons is largely unnecessary.

As agentic systems creep closer to production use—from industrial robots to AI-driven autonomous workflows—the lesson is simple:

Plan, act, observe, and revise. And give your VLM a memory—it performs better when it remembers its own intentions.

Cognaptus: Automate the Present, Incubate the Future.