The Model Remembers What the Firewall Forgets

TL;DR for operators

The current enterprise LLM mistake is to treat safety as a chat-interface problem. Put a filter in front, add a stern system prompt, run a cheerful demo, and hope the model has suddenly acquired a moral philosophy and a compliance department. Charming. Also insufficient.

Two recent papers make a stronger and more operationally useful point. The first evaluates open-source LLMs under prompt-injection and jailbreak attacks, then compares lightweight inference-time defenses such as input filtering, system-prompt hardening, vector detection, voting, and self-examination.¹ The second studies whether fine-tuned language models memorize sensitive personally identifiable information that appears only in inputs, not in the desired outputs, and benchmarks mitigation methods such as differential privacy, UnDial, regularization, and DPO.²

Read together, they form a lifecycle-risk chain:

Sensitive data enters fine-tuning
        ↓
The model internalizes some of it
        ↓
Adversarial interaction probes weak points
        ↓
Lightweight guardrails reduce some failures
        ↓
Residual leakage, silent refusal, or brittle behavior remains

The business conclusion is blunt: prompt filters are not a safety architecture. They are one control in a larger system. Enterprises need to manage LLM risk across data intake, fine-tuning, model selection, runtime defenses, refusal behavior, red-team testing, logging, and incident response. Otherwise the organization merely moves risk from the training pipeline to the chat window, where it waits politely until someone asks the wrong question with enough theatrical formatting.

Why this matters now

Enterprise AI adoption is moving toward smaller, cheaper, locally deployable, and task-adapted models. That shift is rational. Open-source or locally hosted models can reduce API dependency, support data-residency requirements, and make domain-specific automation more practical. They also tempt organizations into a comforting story: because the model is “ours,” and because the deployment is “private,” the risk is under control.

The papers examined here disturb that story from two directions.

One paper looks outward, at the interaction layer. It asks how open-source models behave when users try to override instructions, role-play around restrictions, or bury adversarial intent inside long prompts. It also tests defenses that many organizations actually reach for because they do not require retraining.

The other paper looks inward, at the training layer. It asks whether sensitive data can become embedded in a fine-tuned model even when that data was never supposed to be generated. This is the awkward privacy problem hiding behind many domain adaptation projects: the model can learn more than the task asks it to learn.

The useful article is not “Paper A says X, Paper B says Y.” That would be a filing cabinet, not analysis. The useful structure is a chain. The memorization paper explains how sensitive liabilities can be created upstream. The jailbreak paper explains how downstream interaction and guardrail weakness can expose or amplify unsafe behavior. Together, they point to a practical governance lesson: LLM safety is not a wrapper. It is a control system.

The shared problem: LLMs are not passive software components

Traditional enterprise software usually fails inside a relatively crisp boundary. A database leaks because access control fails. An API misbehaves because validation is poor. A workflow system breaks because a rule was misconfigured. These failures can be serious, but the component does not usually reinterpret instructions, infer missing context, summarize confidential records, role-play as a different component, or produce plausible nonsense while looking helpful.

LLMs do all of those things by design. That is the product feature, inconveniently wearing a risk costume.

The two papers study different sides of that feature. The prompt-injection paper focuses on models under adversarial user interaction. Its key question is whether open-source models resist attacks that exploit instruction-following itself. The memorization paper focuses on fine-tuning, where the model may absorb sensitive details from inputs even when the downstream task only requires classification, extraction, or planning.

The bridge between them is this: a model’s deployed behavior is not created only at inference time. It is shaped by pretraining, fine-tuning, alignment, prompt context, runtime defenses, and user pressure. When those layers are governed separately, the organization gets fragmented controls. Fragmented controls are excellent at producing dashboards. They are less excellent at stopping failures.

What the papers show, in relationship

This is the central synthesis: one paper explains how the model can acquire a privacy liability; the other explains how an attacker or careless user can later pressure the deployed system into revealing unsafe behavior. The papers do not prove that every memorized PII item will be exposed through every jailbreak. They do something more useful for operators: they show why upstream data exposure and downstream prompt robustness belong in the same risk model.

Step 1: Fine-tuning can make sensitive data sticky

The memorization paper isolates a subtle but business-critical case: personally identifiable information appears in the fine-tuning input, but not in the desired output. For example, a clinical or financial document may contain names, dates, addresses, or contact details, while the training target asks for classification, information extraction, or follow-up planning. The model is not being trained to output the private details. The private details are merely present. Unfortunately, “merely present” is doing a lot of work.

The authors define this as unintended, input-only PII memorization. They evaluate it using synthetic multilingual financial data and private German medical datasets, including pathology reports and discharge summaries. Their attack setting is deliberately relevant to deployment: adversaries have black-box access through model queries and may have partial information from the fine-tuning corpus. The main memorization probe is a True-Prefix Attack, where the model is given text immediately preceding a PII span and assessed on whether it reproduces the sensitive continuation. They also use an enhanced version that adds the first character of the PII item.

The paper’s findings are not reducible to “fine-tuning bad.” Fine-tuning improves downstream task performance, which is precisely why businesses do it. The problem is that task adaptation can increase the model’s confidence in predicting sensitive tokens. The authors show that fine-tuned models become more confident on PII tokens, and that memorization depends on factors such as language, downstream task, prefix length, model capacity, and context. Frequency alone is a poor predictor of what gets memorized; the paper reports that PII repetition explains less than 24% of the variance in extraction. That matters because many compliance workflows still lean on crude heuristics: remove frequent obvious identifiers, assume the rest is low risk, proceed to the innovation slide deck.

The business interpretation is not that enterprises must never fine-tune on sensitive data. That would be simple, elegant, and frequently impossible. The interpretation is that fine-tuning creates an exposure channel that must be tested directly. If a model sees sensitive material during adaptation, the organization should assume that some of it may become recoverable unless proven otherwise under realistic extraction tests.

That shift is important. Privacy review cannot stop at “PII is not in the target labels.” For LLMs, the input is not inert. It is training material.

Step 2: Adversarial prompts exploit helpfulness, not just keywords

The prompt-injection paper examines the deployed interaction layer. It evaluates open-source models ranging from roughly 1B to 7B parameters, including variants from Llama, Mistral, Phi, Qwen, Gemma, and DeepSeek. The authors use prompt sources from communities, repositories, benchmarks, research papers, and manually crafted variants. The evaluated attacks include question-answering setups with hidden adversarial intent, basic compliance tests, instruction overrides, role-playing jailbreaks, and long-format or multi-step prompts.

The first operator lesson is that attack form matters. Simple keyword-style filters may catch blunt prompts. But sophisticated prompts do not need to announce themselves with a neon sign saying “jailbreak.” They can be long, contextual, polite, framed as fiction, staged as multi-role dialogue, or structured as a reasoning task. The paper’s defense evaluation emphasizes that longer, contextually rich prompts are harder to defend against. Some popular jailbreaks were weak or outdated, while stronger multi-role and override-style prompts remained more effective. This is exactly how adversarial behavior evolves in production: the low-effort attacks become benchmark nostalgia; the better ones learn to dress for the occasion.

The second lesson is that refusal behavior is itself part of system quality. In baseline prompt-injection testing, some models showed zero vulnerability but did so through silent non-responsiveness: empty outputs, no refusal explanation, and little debuggability. That is not the same as a robust, transparent refusal. From a business standpoint, silent failure is still failure. It may be safer than harmful output, but it is operationally ugly. Users cannot tell whether the request was blocked, the model crashed, the policy fired, or the system went out for coffee.

The paper’s defense results are also instructive. Self-examination performed best among the tested inference-time defenses, while input filtering was consistently weakest. System-prompt hardening helped but remained inconsistent. Vector and voting defenses reduced some vulnerability but did not eliminate failures, and they introduced their own operational costs such as latency, threshold tuning, or multiple generations.

This is where the two papers connect. If fine-tuning can create latent sensitive memorization, then the interaction layer becomes the place where that liability may be probed. The specific attack mechanisms differ: true-prefix extraction is not the same as a jailbreak role-play. But both rely on the same unpleasant property: model behavior can be steered by context, partial information, and carefully structured prompts.

Step 3: The control layer is not one control

The tempting enterprise response is to ask, “Which defense should we buy?” This is understandable. It is also too small a question.

The papers show that defenses are conditional. In the prompt-injection paper, input filtering is cheap and fast but brittle. System-prompt hardening is transparent and easy to update but depends on the model respecting the instruction hierarchy. Vector defenses generalize beyond exact keywords but require good embeddings, a representative attack database, and careful thresholding. Voting improves robustness in some cases but increases inference cost. Self-examination can catch subtle unsafe outputs, but it adds latency and may itself be vulnerable or inconsistent.

In the memorization paper, mitigation methods are also conditional. Differential privacy can strongly reduce leakage in some settings, but it is sensitive to hyperparameters and can affect utility. UnDial and DPO can offer favorable privacy-utility trade-offs, but performance depends on setup, model size, task, and data. Regularization is not automatically superior. Combining approaches does not necessarily produce the best of both worlds. The authors explicitly find that even strong methods leave residual leakage; in one summary, the most effective methods achieved only around a 40% reduction in direct PII memorization.

The common pattern is not “Defense X wins.” It is “defense performance is context-dependent, incomplete, and measurable.” That is the sentence procurement teams should tape to the wall before anyone buys a guardrail product with a dashboard and a heroic font.

A useful enterprise control model therefore looks like this:

Notice the structure. These controls do not replace one another. They cover different points in the chain. A runtime filter cannot retroactively make fine-tuning data safe. Differential privacy cannot decide whether a live user is attempting a prompt injection. A system prompt cannot prove that PII was not memorized. Each control answers a different question.

Step 4: Model size is not a governance strategy

Both papers complicate the lazy scaling story.

In the prompt-injection paper, larger variants were not automatically safer. The authors report non-monotonic safety patterns: smaller variants of a model family could be much more vulnerable than larger ones, but larger models could also remain vulnerable under weak defenses, likely because stronger instruction-following can become a better attack surface when safety does not scale with capability. Alignment strategy and refusal design matter more than parameter count alone.

In the memorization paper, scaling also changes the privacy trade-off. Larger models can improve task performance, but the mitigation that works best at one size may not behave the same way at another. Differential privacy was strong in some smaller-model settings, while DPO became more effective for a larger model in the authors’ scaling analysis. The direction is not “bigger leaks” or “bigger safe.” The direction is “bigger changes the control problem.”

That has a direct business implication. Model evaluation should be deployment-specific. A vendor’s general safety claim is not enough. A leaderboard score is not enough. An internal benchmark on ordinary task performance is not enough. The model should be tested under the organization’s data conditions, expected user behavior, adversarial prompts, latency constraints, and incident-response requirements.

A model can be very capable and still unsafe for a particular workflow. A model can refuse harmful content and still be unusable because it silently fails. A model can pass a friendly test and still leak under a prefix-style probe. The point is not to distrust all models equally. The point is to stop outsourcing risk judgment to model size.

What operators should measure

The combined lesson is measurable. That is good news. Governance becomes much less theatrical when it asks for evidence.

This table is not meant to be decorative. It is a procurement and governance checklist. If a team cannot answer these questions, it should be careful about claiming the model is “safe for enterprise.” It may be useful. It may be promising. It may even be less risky than the manual process it replaces. But “safe” requires evidence.

The misconception to kill: “We added a prompt filter”

The most dangerous misunderstanding is that LLM safety mainly means catching bad prompts. The prompt-injection paper shows why this is weak: surface-level filters struggle against long, structured, reasoning-heavy attacks. The memorization paper shows why it is incomplete: the model may already contain sensitive liabilities before the user prompt arrives.

A prompt filter is like a security guard at the lobby desk. Useful, visible, reassuring. But if confidential files have already been photocopied into the building’s walls, and if visitors can talk the elevator into opening restricted floors, the lobby guard is not the architecture. It is scenery with a badge.

The right question is not “Do we have a guardrail?” It is:

Which failure mode does this guardrail reduce, how was that measured, what does it miss, and what happens when it fails?

That question should be asked for every control in the chain.

A practical operating model

Here is a compact way to translate the papers into an enterprise control design.

1. Classify the deployment by exposure, not enthusiasm

Before fine-tuning or deployment, classify the workflow:

• Does the model see regulated, confidential, or personal data?
• Does the model generate outputs for external users, employees, or downstream systems?
• Can users supply arbitrary prompts?
• Does the model use retrieval, tools, email, files, or workflow actions?
• Would leakage cause legal, financial, safety, or reputational harm?

A customer-support classifier with no sensitive data and no free-form output is a different risk category from a clinical summarizer fine-tuned on identifiable records and exposed through a chat interface. This should not require a committee retreat. It requires discipline.

2. Test privacy before testing charm

Fine-tuned models should be tested for extraction, not just accuracy. The memorization paper’s broader point is that task performance can improve while privacy risk remains. Operators should compare at least three states:

Base model
Fine-tuned model
Fine-tuned model with mitigation

For each state, measure task utility and extraction risk. If mitigation improves privacy but destroys the task, that is not deployment-ready. If it preserves the task but barely reduces leakage, that is also not deployment-ready. The metric is the trade-off, not the slogan.

3. Red-team the interaction layer with realistic prompts

Red-team prompts should include role-play, instruction override, multi-step framing, long-form prompts, and domain-specific extraction attempts. They should not be limited to famous old jailbreaks copied from the internet. The prompt-injection paper’s finding that some popular attack sources were weak or outdated is a useful warning: benchmark comfort can become historical reenactment.

The attack suite should be refreshed. Attackers, unlike governance decks, do update.

4. Record failure modes, not only pass rates

A binary safe/unsafe score is too thin. The papers surface important qualitative distinctions:

• harmful compliance,
• partial compliance,
• refusal with explanation,
• silent non-response,
• timeout or instability,
• leakage under sampling but not greedy decoding,
• leakage of exact versus approximate PII,
• leakage of distinct versus repeated identifiers.

These distinctions matter operationally. A silent refusal might reduce immediate harm but create support tickets and debugging darkness. A partial answer may be more dangerous than a full refusal because it leaks enough to be useful. Approximate PII may still identify a person. A model that fails only under sampling may still fail in production if generation settings differ from test settings.

5. Assign ownership across the chain

The data team owns source exposure. The ML team owns fine-tuning behavior. The security team owns adversarial testing. The product team owns user experience around refusal. Legal and compliance own regulatory interpretation. Operations owns monitoring and incident response.

No single team owns the whole chain by default. That is the problem. The control system has to make ownership explicit, because otherwise every failure becomes “somebody else’s layer.”

What not to overclaim

The papers are useful, but they do not justify universal conclusions.

The prompt-injection study focuses on specific open-source models, prompt sets, and lightweight defenses. Its results should not be treated as a permanent leaderboard of safe and unsafe models. Model versions change, attack prompts evolve, and deployment architecture matters. It also notes execution instability in some configurations, which is itself operationally relevant but limits neat comparison.

The memorization study focuses on parameter-efficient fine-tuning, specific model families, and datasets with particular properties. The authors explicitly note limitations around larger foundation models, reasoning-oriented architectures, full fine-tuning, dataset availability, and PII annotation quality. Its strongest business lesson is not a universal leakage percentage. It is the evidence that input-only sensitive data can be memorized and that mitigation remains incomplete.

That boundary matters. Good strategy does not pretend two papers settle LLM safety. It uses them to improve the questions businesses ask before deployment.

The combined conclusion

The two papers sit on different sides of the same enterprise problem. One shows that sensitive information can become embedded in a model through fine-tuning even when the task target does not require it. The other shows that deployed models remain vulnerable to adversarial interaction and that lightweight defenses reduce, but do not eliminate, prompt-layer failures.

Together, they argue for a lifecycle view of LLM risk:

Do not only ask: "Will the model answer this bad prompt?"
Also ask:       "What did the model absorb before the prompt arrived?"

That is the operational shift. LLM governance cannot be a decorative filter taped to the front of a model. It has to connect data policy, fine-tuning practice, model evaluation, guardrail design, refusal behavior, red-team testing, and runtime observability.

The uncomfortable lesson is that enterprises may need to slow down at exactly the points where AI adoption feels easiest: copying data into a fine-tuning job, selecting a small open-source model because it runs locally, adding a guardrail wrapper, and calling the system production-ready. These are not reasons to avoid deployment. They are reasons to deploy with evidence.

A model that remembers what it should not, and then can be talked around its guardrails, is not an intelligent assistant. It is a compliance incident with autocomplete.

Cognaptus: Automate the Present, Incubate the Future.