When Prompt Optimization Overloads the Small Model

TL;DR for operators

A small model has a finite inference budget. Prompt optimization consumes that budget too: longer instructions occupy context, optimization calls add latency, and extensive rewriting can alter an input the model might already have understood.

Shim and colleagues test a different operating rule in POaaS: inspect each query first, leave sufficiently good prompts alone, and apply narrowly targeted repairs only when a specific deficiency is detected.¹ On Llama-3.2-3B, this raises average clean task accuracy from 63.7% with no optimization to 66.0%. Under the same fixed-small-model protocol, EvoPrompt, OPRO, and PromptWizard fall to 59.8%, 57.5%, and 48.8%.

For teams deploying edge or on-device assistants, the relevant decision is not simply whether to optimize prompts. It is how much context, latency, and model capacity the optimizer is allowed to consume. POaaS makes a strong benchmark case for selective repair under tight budgets. It does not establish that search-based automatic prompt optimization is generally inferior, nor does it demonstrate gains on production devices.

More optimization can leave less capacity for the actual task

Suppose an on-device assistant already runs close to its context and compute limits. A prompt optimizer that produces a sophisticated multi-thousand-token instruction may improve the wording presented to the model, yet also create a harder inference environment: more tokens must be processed, more instructions must be reconciled, and the same limited model may be asked to participate in generating the optimization itself.

That tension appears directly in the paper’s main comparison.

On Llama-3.2-3B, the unoptimized baseline averages 63.7% across BBH, GSM8K, and CommonsenseQA. EvoPrompt reaches 59.8%, OPRO 57.5%, and PromptWizard 48.8%. POaaS reaches 66.0%. The pattern repeats on Llama-3.1-8B: no optimization scores 70.1%, the three search-heavy baselines score between 54.9% and 66.1%, and POaaS reaches 71.5%.

Factuality moves in the same direction. POaaS raises the three-dataset average from 44.5% to 48.1% on the 3B model and from 51.9% to 53.3% on the 8B model. Each evaluated search-based baseline finishes below the corresponding unoptimized condition.

These are the paper’s main comparative results, not evidence that elaborate prompt optimization fails in general. The protocol deliberately requires optimizer, critic, evaluator, and task inference inside each APO baseline to use the same 3B or 8B backbone. The result therefore addresses a specific deployment question: what happens when prompt optimization must live within the capacity envelope of the small model it serves?

Under that condition, complexity is not free.

POaaS decides whether a prompt should be touched at all

POaaS changes the optimization problem from global prompt redesign to bounded per-query intervention.

A CPU-side orchestrator first scores properties including typos, completeness, fluency, and clarity. Inputs that appear sufficiently well formed can bypass refinement entirely. Inputs with detected problems are routed to one or more LoRA-based specialists: Cleaner for error correction, Paraphraser for semantic reformulation, and Fact-Adder for limited factual priming.

The system then constrains what those specialists are allowed to change. Candidate edits face lexical-similarity checks, content-preservation requirements, structural safeguards, answer-leakage filters, and a prompt-expansion limit. Fact-Adder is capped at three short bullets and 120 tokens. Failed candidates fall back to the original prompt.

This matters because automated rewriting introduces its own failure mode. An optimizer can make prose more polished while changing numbers, entities, quotations, URLs, or the task itself. POaaS explicitly measures this change as lexical drift and penalizes edits that fail to retain key content.

The architecture therefore spends optimization capacity selectively. Detection happens cheaply on CPU; specialist model calls are invoked only when routing indicates a problem; and accepted modifications have explicit budgets.

That is materially different from treating every request as an opportunity to synthesize a better global instruction.

Corrupted prompts reveal what the specialists actually do

The paper’s token-deletion and token-mixup experiments are robustness tests, not a second benchmark thesis. Their purpose is to ask whether the repair architecture becomes more useful when input quality deteriorates.

It does.

At 15% token deletion, the unoptimized 3B model falls to 40.2% average task accuracy. POaaS reaches 47.6%, a 7.4-point improvement. On the 8B model, it raises accuracy from 45.6% to 52.8%, or 7.2 points.

With 15% token mixup, the gains are 5.8 points for 3B and 3.8 points for 8B. Meanwhile, the search-heavy baselines deteriorate progressively as corruption increases.

Table 4’s component ablations clarify why. Removing Cleaner leaves most clean-input improvement intact but nearly eliminates the recovery under corrupted inputs. On the 3B model at 10% deletion, full POaaS scores 52.8%, versus 45.8% with no optimization. Removing Cleaner drops performance to 46.4%.

Fact-Adder behaves differently. On clean 3B inputs, Fact-Adder alone raises accuracy from 63.7% to 65.6%, while Cleaner alone reaches only 63.9%. Similar separation appears on the 8B model.

The specialists are therefore not interchangeable sources of generic “agent” improvement. Cleaner repairs damaged inputs. Fact-Adder contributes more strongly when the input is already structurally sound but limited contextual priming can help. Paraphraser produces smaller gains across both regimes. The full configuration performs best because the routing layer can apply different interventions to different deficiencies.

The efficiency numbers require an apples-to-apples reading

In the 3B experiment, POaaS averages 0.95 seconds of refinement, 1.4 specialist calls, and 48 added prompt tokens per query.

The APO configurations report 336 to 602 seconds of optimization, 96 to 172 internal calls, and static instructions adding 2,840 to 4,180 tokens.

The token difference directly illustrates the context-budget problem. The latency comparison needs more care. APO optimization time is measured as offline search used to produce a static instruction, whereas POaaS’s 0.95 seconds is incremental online latency paid per query. Those numbers describe different cost structures.

For a product team, the choice therefore depends partly on workload shape. A reusable static prompt can amortize offline search across many requests. POaaS instead avoids a large instruction payload but charges selective refinement at inference time.

Cognaptus’s operational inference is to benchmark both architectures in the deployment regime that will actually be used: production context lengths, expected query volume, tolerated latency, available accelerators, and the same model capacity assigned to optimization.

Production deployment remains an open test

The study provides controlled comparative evidence, not field evidence from phones, laptops, embedded devices, or production assistants.

Its specialists and heuristics are English-only. Routing thresholds and caps are empirically tuned. Only two Llama instruction-tuned backbones and three specialist roles are evaluated. Factuality is measured using GPT-5 as a binary judge rather than direct human assessment. The target model is frozen, but POaaS is also not training-free: its Cleaner, Paraphraser, and Fact-Adder components are fine-tuned LoRA adapters.

Those boundaries matter most when transferring the architecture. A multilingual assistant, a domain with fragile terminology, or a smaller model may require different routing signals, preservation tests, and specialist training. A production device may also value 0.95 seconds of additional latency very differently from a benchmark harness.

What carries forward is a testable deployment principle. Prompt optimization should be evaluated as part of the model’s resource envelope. If the optimizer consumes context, compute, or representational capacity that the solver also needs, increasing optimization complexity can reduce downstream performance.

POaaS shows that a smaller intervention surface can work better under those constraints: detect the failure, route it to the relevant repair mechanism, constrain the edit, and leave good prompts alone.

Cognaptus: Automate the Present, Incubate the Future.