Prompted and Confused: When LLMs Forget the Assignment

A requirements document walks into a model.

It says: assign resources, respect capacity, avoid conflicts, minimise waste. The model nods politely, emits a tidy block of MiniZinc, and everyone is briefly tempted to believe the future has arrived. Then someone changes the story from cars to knapsacks, or adds one stray sentence about maximising something, and the same system quietly forgets the assignment.

That is the useful discomfort in Alessio Pellegrino and Jacopo Mauro’s paper, “When Words Change the Model: Sensitivity of LLMs for Constraint Programming Modelling.”¹ The paper does not ask whether large language models can generate constraint programming models under ideal benchmark conditions. They often can. The sharper question is whether they still generate the right model when the wording changes but the underlying combinatorial structure does not.

The answer is: sometimes, and not nearly reliably enough to treat the output as understood.

This matters because constraint programming is not decorative coding. It is how organisations express hard operational structure: schedules, allocations, packing problems, routing rules, compatibility constraints, workforce assignments, manufacturing sequences. A constraint model can be syntactically neat and semantically wrong. That is a particularly elegant failure mode, which is to say: the dangerous kind.

The real test is not generation, but invariance

The paper’s central move is simple and effective. It takes well-known constraint programming problems, mostly from CSPLib, and asks whether LLMs can model them in MiniZinc under three forms of description.

First, the models receive the original problem descriptions. These are canonical, familiar, and plausibly present somewhere in the public training data soup. Second, the authors rewrite the context while preserving the same mathematical structure. Car sequencing becomes knapsack assembly; Sudoku becomes a chess-like placement problem; vessel loading becomes arranging paintings in a gallery. Third, they add distracting or misleading information while still intending the same underlying problem.

That third condition is where the trap becomes commercially recognisable. Real business requirements are rarely written like competition benchmarks. They include background noise, preferences that are not constraints, objectives mixed with anecdotes, operational caveats, and executive prose that has seen things. The model must decide what is structural and what is merely narrative.

The paper evaluates three LLMs: GPT-4, Claude 4, and DeepSeek-R1. Each gets one attempt, no examples, the same system prompt, temperature set to zero, and a requirement to produce MiniZinc. The authors assess outputs using two metrics: whether the generated model is conceptually correct, and whether it runs successfully in MiniZinc with only minor manual fixes.

The result is not “LLMs cannot do constraint programming”. That would be too easy, and also false. The result is more awkward: LLMs can perform impressively on familiar formulations, then degrade when the surface language stops matching the template.

The keyword that hijacks the model

The most revealing mechanism is lexical hijacking. A word such as “maximise” can pull the model away from the actual problem structure and into the wrong modelling frame.

In the Sudoku example, the original puzzle is a satisfaction problem: fill the grid so each row, column, and subgrid contains the right values. The authors reframe it as placing queens with different garments on a chessboard. With only the context changed, the models still cope. GPT-4 even recognises the Sudoku-like structure.

Then the distractor introduces wording about maximising the number of garments while avoiding attacks. The underlying structure is still intended to mirror the satisfaction problem, but the word “maximise” changes the models’ interpretation. They turn the task into an optimisation problem.

That is not a small coding bug. It is a category error.

The same pattern appears elsewhere. In the Perfect Square problem, the task is to pack smaller squares into a larger square with no unused space. When the distractor says the park design should minimise unused areas, all three LLMs reformulate the problem as optimisation. Again, the surface word overpowers the deeper structure.

This is the first lesson for business use: prompts are not informal conversation when the output is a formal model. They are specifications. A single lexical cue can become a false hard constraint. The model is not reading the room; it is weighting tokens. Occasionally, those tokens have terrible judgement.

Context changes expose where the template was doing the work

Context rewriting is the paper’s way of separating structural understanding from recognition.

If a model genuinely understands the car sequencing problem, it should not matter too much whether the story is about cars moving through production or knapsacks with accessories being assembled under capacity limits. The variables may change names, but the demand structure and capacity constraints should survive.

They do not.

In the modified Car Sequencing problem, all three LLMs fail. The key error is that they treat demand as a single global quantity rather than as a type-specific demand array. That mistake does not appear in their responses to the original formulation. The most plausible interpretation is not that knapsacks are inherently harder than cars. It is that the canonical car-sequencing wording activates a known modelling pattern, while the rewritten context removes the crutch.

The Secret Shopper problem shows a related fragility. In the original version, the task involves scheduling visits so each worker is visited four times by distinct shoppers, with additional constraints. After the authors replace workers with law enforcement officers and remove some redundant numerical details, all three models miss the constraint requiring each worker to be visited by different officers. The paper suggests that the removed numerical scaffolding may have helped the models preserve the right structure.

This is where the “LLMs reason over requirements” story needs editing. Sometimes they do. Sometimes they recognise a familiar benchmark outfit and behave accordingly. Change the outfit, and the modelling competence looks less like abstraction and more like memory with a blazer.

Distractors turn helpfulness into modelling pollution

The distractor tests are not just robustness checks; they are a proxy for messy requirements. They ask whether the model can ignore information that sounds operationally relevant but is not actually part of the formal problem.

The answer is mixed, and the failures are instructive.

In Template Design, the authors add a statement about occasional misalignment in the cutting process. The sentence does not define a formal constraint. Claude 4 and GPT-4 nevertheless add extra constraints for misalignment. DeepSeek-R1 explicitly declines to model it because no specific constraints are provided. For once, the less eager answer is the better engineering answer.

In Social Golfers, the distractor introduces research areas and suggests that teams should ideally be interdisciplinary. All three LLMs add a new interdisciplinarity constraint, even though it is not part of the stated objective. In Golomb Ruler, a distractor about beacon ranges and costs causes GPT-4 and Claude 4 to incorporate cost-related constraints, while DeepSeek-R1 preserves the original formulation and explicitly relates it to the Golomb ruler structure.

This is a familiar failure in enterprise automation. The system tries to be helpful by incorporating every plausible business sentence. Unfortunately, formal modelling is not a brainstorming session. If a requirement says “ideally”, “where possible”, or “stakeholders prefer”, the modelling pipeline must decide whether that becomes a hard constraint, a soft constraint, an objective term, a reporting note, or nothing at all.

The paper’s distractor results show that current LLMs may collapse these categories. That collapse is costly because it can produce a model that is not obviously broken. It may run. It may look sophisticated. It may even contain more constraints than the baseline. Excellent. The machine has made the wrong problem more elaborate.

Mathematical anchors help, but they are not a personality transplant

The strongest positive result is also the most practically useful: explicit mathematical structure improves robustness.

Problems such as Quasi Group Existence and All-Interval Series remain comparatively stable under rewording and distractors. In Quasi Group Existence, all three LLMs recognise the Latin-square-like structure even after the context is changed to delivery-drone scheduling. When a workload-balance distractor appears, the models largely ignore it and preserve the core structure.

The Low Autocorrelation Binary Sequence problem shows the same pattern. The task includes an explicit objective function for minimising autocorrelation. Even when the authors rewrite it as a music composition task and add a distractor about consecutive beats reinforcing each other, GPT-4 and DeepSeek-R1 continue to follow the explicit formula. Claude 4 adds an unnecessary balance constraint, so the anchor is not magic. But it clearly helps.

This is a useful correction to the lazy prompt-engineering slogan. The lesson is not “write clearer prompts”, though one may print that on a mug if one has abandoned hope. The lesson is more specific: when asking LLMs to produce formal optimisation models, provide explicit variables, domains, constraints, and objectives wherever possible. Mathematical anchors reduce the interpretive surface area.

A practical structure might look like this:

The point is not to replace modelling expertise with algebraic theatre. It is to give the model less room to hallucinate structure from prose.

The benchmark contamination problem is not a footnote

The paper also examines possible data contamination. Since the authors cannot inspect proprietary training data, they use an indirect test: provide the first half of a problem description and ask the LLM to complete the second half. If the completion closely matches the original description in objective, framing, and structure, that suggests prior exposure.

The results are not definitive proof, and the authors do not pretend otherwise. But they are suggestive. Claude 4 completes five of the ten CSPLib descriptions in a way the authors consider close to the originals. GPT-4 completes four. DeepSeek-R1 completes six, although the paper notes that at least one may be a generic completion rather than clear evidence of memorisation.

This matters because strong benchmark performance on canonical descriptions can be inflated by recognition. If an LLM has seen the problem, seen the MiniZinc model, or seen nearby formulations, it may produce something that looks like reasoning but is partly retrieval.

For businesses, the contamination issue has a slightly different shape. The risk is not merely that public benchmark scores are optimistic. It is that a pilot project built on toy or canonical examples may overstate readiness for internal requirements. The demo problem is recognisable; the real problem is written by operations, finance, compliance, and three people who were copied in too late. Different sport.

The numbers show degradation, not collapse

The quantitative table in the paper is small, but it gives the qualitative analysis a useful backbone.

Across the eleven total tasks, the original descriptions produce high conceptual correctness: Claude 4 gets 11 out of 11 correct, GPT-4 gets 9, and DeepSeek-R1 gets 10. Executability is weaker: Claude 4 and DeepSeek-R1 each produce 6 runnable models, while GPT-4 produces 4.

With context modifications only, conceptual correctness declines. Claude 4 and DeepSeek-R1 each get 8 correct, while GPT-4 gets 6. Runnable outputs are 7, 2, and 6 respectively. The odd detail that Claude’s runnable count is higher under context modification than under original descriptions is a reminder not to overinterpret each cell in a small manual study.

The steepest decline appears when both context changes and distractors are introduced. Correct outputs fall to 2 for Claude 4, 3 for GPT-4, and 5 for DeepSeek-R1. Runnable outputs fall to 2, 1, and 4 respectively. None of the models correctly formulates even half of the distractor variants.

A compact way to read the study is this:

The right conclusion is not panic. It is also not procurement-friendly optimism. The study shows degradation under controlled perturbation. That is precisely the kind of evidence organisations should want before they hand LLMs a warehouse schedule and a budget.

Correct code is not the same thing as the right model

The paper’s distinction between “correct” and “run” is important because it prevents two common mistakes.

The first mistake is treating compilation as correctness. A MiniZinc model can run while solving the wrong problem. The second mistake is treating non-execution as the only risk. A model may be semantically close but fail because of repairable syntax, missing imports, or output formatting. Both signals matter, but neither is sufficient alone.

For operational use, this implies a layered validation pipeline.

First, parse the generated model for syntax and execution. This catches the easy failures. Second, test it on small known instances where expected outputs are obvious. Third, mutate the requirements: paraphrase the problem, remove irrelevant wording, add distractors, and check whether the generated structure remains stable. Fourth, have a domain expert review whether each constraint corresponds to an actual business rule. Fifth, where possible, compare generated solutions against historical decisions or hand-built baselines.

None of this sounds as glamorous as “natural language to optimisation”. That is because glamour is usually the first thing to break in production.

The business value is assisted modelling, not unattended modelling

The paper’s strongest business implication is not that LLMs are useless for optimisation. It is that their most credible near-term role is assisted modelling under verification.

An LLM can help draft candidate formulations, translate informal descriptions into variable and constraint candidates, explain modelling alternatives, and generate MiniZinc scaffolding. Those are valuable tasks. They reduce friction for teams that have operational knowledge but limited modelling capacity.

But the study makes unattended modelling look fragile. A system that can be pulled off course by a context shift or a stray objective-like phrase should not be the final authority on capacity planning, crew scheduling, inventory allocation, vehicle routing, or production sequencing.

The practical pathway looks like this:

This is where Cognaptus would draw a fairly unsentimental line. The ROI is not “replace optimisation specialists”. The ROI is “make specialists faster, make requirements more explicit, and catch modelling misunderstandings earlier”. Less cinematic, more useful.

The limits are specific, and that is why the paper is useful

The study is not a universal verdict on all LLM-based optimisation modelling. Its scope is deliberately narrow: ten CSPLib problems plus Sudoku, three LLMs, English descriptions, one-shot prompting, MiniZinc output, manual evaluation, and indirect contamination evidence.

Those boundaries matter.

A multi-turn workflow might perform better. A system that asks clarifying questions could reject ambiguous distractors. A solver-in-the-loop pipeline could repair syntax, test instances, and flag inconsistent constraints. Retrieval over a curated modelling library might improve structure. Different models, newer models, different temperatures, or different modelling languages could change the results.

The paper also relies on manually designed reformulations. The authors aim to preserve structural equivalence, but some rewritten prompts may introduce ambiguity or difficulty beyond pure context variation. That does not invalidate the finding; it simply narrows it. The paper shows a real fragility under plausible perturbations, not an exhaustive theory of LLM reasoning.

And that is enough.

In business settings, narrow evidence of a plausible failure mode is valuable. You do not need a grand unified theory of model brittleness to decide that auto-generated constraint models need review.

The prompt is part of the system architecture

The most useful way to read this paper is not as an anti-LLM argument. It is an argument against a sloppy boundary between language and formal systems.

When natural language becomes the interface to optimisation, the prompt is no longer just a request. It is part of the system architecture. It shapes variables, constraints, objectives, and omissions. If it contains ambiguity, the model may resolve that ambiguity silently. If it contains distractors, the model may formalise them. If it resembles a benchmark, the model may retrieve a familiar pattern and look smarter than it is.

That does not make LLMs unusable. It makes governance boringly necessary.

The future of LLM-assisted optimisation will probably not be a manager typing “schedule everyone efficiently” into a box and receiving truth. It will be structured intake, formal anchors, generated candidate models, solver checks, adversarial paraphrases, and human review. A little less magic, a little more engineering. Tragic, but survivable.

The models in this paper did not fail because they could not write code. They failed because they sometimes wrote the wrong mathematics in perfectly confident prose-adjacent clothing.

That is the assignment businesses need to remember.

Cognaptus: Automate the Present, Incubate the Future.