Prompted and Confused: When LLMs Forget the Assignment

Opening — Why this matters now

The industry narrative says LLMs are marching confidently toward automating everything from tax audits to telescope alignment. Constraint programming — the backbone of scheduling, routing, and resource allocation — is often portrayed as the next domain ripe for “LLM takeover.” Just describe your optimisation problem in plain English and voilà: a clean, executable model.

The paper I reviewed flips that optimism on its head. It reminds us that LLMs can appear brilliant right up to the moment you rephrase a sentence, add a distracting clause, or swap cars for knapsacks. Then the magic crumbles.

Background — Context and prior art

Constraint Programming (CP) is exacting. Real solutions depend on understanding structural rules, leveraging global constraints, and selecting representations that avoid computational disasters. Prior papers showed LLMs performing suspiciously well — prompting concerns about data contamination. After all, most classical CP tasks live freely on the internet.

This study from Pellegrino & Mauro tests precisely that: can LLMs truly reason about CP tasks, or are they just regurgitating training‑set artefacts?

The authors select the first ten CSPLib problems — well-known, heavily replicated benchmarks — and introduce two types of perturbations:

• Context modification: same underlying problem, new narrative skin.
• Distraction modification: irrelevant or misleading goals added to the prompt.

The results are not flattering.

Analysis — What the paper does

Three models were tested under identical conditions: GPT‑4, Claude 4, and DeepSeek‑R1. No example-based prompting, no iterative refinement — just raw zero‑shot modelling into MiniZinc.

Each problem was fed in three versions:

1. Original description
2. Context-modified (same structure, different story)
3. Context + distraction (classic real-world ambiguity)

The authors then assessed two simple things:

• Correctness: does the model capture the intended constraints?
• Executability: does the MiniZinc code actually run?

They also conducted a clever contamination check: feeding models only the first half of a problem description and asking them to “finish” it. If the continuation resembles the canonical problem text, that’s a red flag for training-set leakage.

Findings — Results with visualisation

The quantitative results speak clearly.

1. Data contamination is widespread.

Claude completed 5/11 descriptions accurately from half the prompt. GPT‑4 got 4/11. DeepSeek‑R1 scored 6/11.

In other words: these models likely memorised many CP benchmarks.

2. Performance collapses under linguistic perturbation.

Below is a simplified summary inspired by Table 1 (page 5 of the paper) fileciteturn0file0:

The drop is not marginal — it’s catastrophic.

3. Simple wording changes completely redirect the model.

Examples:

• Adding the word “maximize” to a Sudoku variant caused every model to incorrectly treat it as an optimisation task.
• Replacing car assembly with knapsack manufacturing derailed all LLMs even though the underlying sequencing constraints were identical.
• Removing numeric hints from the Secret Shopper task caused all models to omit a key constraint.

4. Mathematical structure stabilises behaviour.

When a problem includes explicit equations (e.g., All‑Interval Series, Quasi‑Group Existence), LLMs almost always stay on track — even under distraction.

This suggests LLMs reason best when the prompt resembles textbook algebra, not narrative description.

Implications — What this means for industry and builders

For businesses contemplating LLM‑generated optimisation models, the lesson is crisp:

LLMs do not yet understand constraints. They recognise patterns. And patterns collapse when the story changes.

This has concrete implications:

1. Zero‑shot modelling is not production‑safe

If your operations team phrases a requirement slightly differently, the resulting model could flip from feasible to nonsensical.

2. LLMs must be paired with verification layers

A hybrid pipeline — LLM proposes, solver validates, and a secondary agent revises — is more realistic.

3. Formal structure is your friend

Feeding LLMs clearer algebraic definitions dramatically improves reliability. Narrative prompts alone are too brittle.

4. Data contamination can inflate benchmarks

Any claimed “reasoning competency” should be treated skeptically without contamination controls.

Conclusion — Wrap-up

This paper delivers a timely reminder: language models are not reasoning engines; they perform linguistic pattern completion. For domains like constraint programming — where nuance matters and misinterpretations carry real operational costs — blind reliance on LLMs is premature.

But with structured prompting, solver‑in‑the‑loop design, and strict validation, LLMs can still serve as useful assistants.

Until then, a healthy skepticism remains the rational stance.

Cognaptus: Automate the Present, Incubate the Future.