Swiss Cheese for Superintelligence: How STACK Reveals the Fragility of LLM Safeguards

TL;DR for operators

Layered safeguards are useful. They are not magic. This paper shows both points, which is inconvenient because the industry prefers safety conclusions that fit on procurement slides.

The authors build and evaluate an open-source defence-in-depth pipeline for LLMs: an input classifier screens the user query, a target model produces an answer, and an output classifier screens the answer before the user sees it. Against ordinary black-box jailbreaks, the best version of this pipeline looks strong. A few-shot-prompted Gemma 2 classifier reduces attack success to 0% on ClearHarm, a dataset focused on clearly harmful catastrophic-misuse queries. That is the good news.¹

Then the paper introduces STACK, a staged attack that treats the pipeline not as one wall but as several separate components. It first works around the input filter, then the model refusal, then the output filter, and combines those component-specific bypasses into one end-to-end attack. Under the paper’s main semi-separable black-box threat model, where an attacker can infer which component blocked a request, STACK reaches 71% attack success on ClearHarm against the same Gemma 2 pipeline that ordinary PAP-style attacks failed to break. In a transfer setting, where the attacker optimises against a proxy pipeline and transfers the attack to the target, STACK reaches 33%.

For operators, the lesson is not “guardrails are useless”. That would be dramatic, comforting, and wrong. The lesson is sharper: guardrail pipelines become weaker when attackers can observe their internal decision boundaries. Refusal wording, latency, separate moderation APIs, streaming behaviour, component-specific error messages, and publicly available models similar to the deployed guard can all become training signals.

The practical response is therefore architectural, not theatrical. Make refusal behaviour less component-specific. Reduce timing side channels. Avoid exposing the same classifiers as separate products where possible. Use diverse guard models rather than near-identical public siblings. Monitor repeated optimisation-like query patterns. Test with adaptive staged attacks, not only standard jailbreak suites. And remember that a 0% attack success rate under a non-adaptive benchmark is not a security guarantee. It is a line item awaiting a better adversary.

The pipeline is not a wall; it is a sequence of decisions

A common mental model for AI safety pipelines is the wall. A user sends a request. The wall decides whether the request is allowed. If not, the wall refuses. Simple. Comforting. Perfect for a diagram with rounded rectangles.

The paper’s more useful model is a sequence. A request first passes an input classifier. If accepted, it reaches the main LLM. If the LLM answers, the output passes another classifier. A harmful result has to survive all three stages: the request must not be flagged, the model must comply, and the response must not be blocked.

That is defence in depth, or the “Swiss cheese” model: each layer has holes, but the holes should not line up. In conventional security, this is sensible. If one layer misses the threat, another layer may catch it. In LLM deployments, this has become the natural architecture for risky capabilities: input moderation, model refusal behaviour, output moderation, usage monitoring, and escalation logic. More slices, fewer holes. At least in the brochure.

The problem is that each slice is also a measurement device. If an attacker can learn which slice blocked the request, the pipeline stops being a single opaque barrier and becomes a diagnostic tool. A failed query tells the attacker what to optimise next. Did the input classifier reject it? Did the model refuse? Did the output filter catch the answer? Each answer narrows the search.

The authors formalise this through threat models. The main one is semi-separable black-box access. The attacker cannot directly inspect weights or query each internal component separately, but can infer where the pipeline failed from observable behaviour. That might be explicit, such as different refusal messages. It might be indirect, such as timing differences. It might come from streaming behaviour, where the system starts producing text and then blocks later. The attacker is still “black-box” in the usual sense. They cannot see the internals. They can, however, hear which door slammed.

That distinction matters for business systems. Many organisations think of moderation as a binary policy function: allowed or refused. The implementation often leaks more than that. A compliance wrapper may return one type of message when a policy classifier blocks the input, another when the foundation model refuses, and another when post-generation screening intervenes. Engineers may treat these as harmless UX details. Red teams treat them as labels.

STACK wins by learning which hole belongs to which slice

STACK stands for STaged AttaCK. The name is slightly too cute, but the mechanism is the point. Instead of trying to defeat the whole pipeline in one clever prompt, STACK decomposes the problem.

At a high level, the attack asks three separate questions:

The paper’s implementation uses PAP, a persuasion-based black-box jailbreak method, for the main model and a custom optimisation method called Confirm for classifier bypasses. The details are less important for most operators than the decomposition. STACK does not need a single universal trick that fools the whole stack. It only needs enough component-level tricks that do not interfere too badly when combined.

That final clause is important. Component-wise attacks can conflict with each other. The paper explicitly observes that STACK can reduce the underlying model-jailbreak effectiveness compared with vanilla PAP on the undefended model. In other words, the combined prompt can become clunky. The attack does not magically improve the model jailbreak; it pays an interference cost. It still wins because the defended system’s classifiers, not the model alone, are the binding constraint.

This is the core mechanism-first insight. STACK is not merely “a better jailbreak”. It is a jailbreak designed for systems with multiple observable failure points. It exploits the fact that a pipeline is easier to attack when the attacker can iterate stage by stage.

That also explains why a normal benchmark can be misleading. A standard black-box attack may fail against a strong pipeline and produce the comforting number every dashboard loves: 0% attack success. But that number only says the tested attack did not line up the holes. It does not say the holes cannot be mapped.

The strongest guard was not the specialised guard model

Before the attack, the paper builds the defence. This is easy to skim past because the attack is more theatrical. Do not skim it.

The authors use Qwen3-14B as the target model and wrap it with input and output classifiers. They evaluate several open-weight safeguard models and also construct few-shot-prompted classifiers from ordinary instruction-tuned models. The surprising result is that the few-shot-prompted Gemma 2 classifier performs best across the baseline attack evaluation, outperforming ShieldGemma, even though ShieldGemma is a specialised moderation model derived from the same Gemma family.

That finding has an awkward operational implication. A model sold, described, or fine-tuned as a safety model is not automatically the strongest guard for a particular misuse boundary. Prompt design, examples, thresholding, and the exact definition of harm can matter as much as the label on the model.

The setup is also favourable to the defender in one crucial way. The authors tune thresholds so pipelines have the same refusal rate on benign inputs: 15%. They note that this refusal rate is impractically high for many applications. Translation: the pipeline is allowed to annoy a lot of normal users, which gives the defender more room to block suspicious traffic. Even under that favourable condition, many open-weight safeguards perform poorly against ordinary attacks.

The key baseline numbers are stark:

These are not production guarantees. They are controlled experimental results on specific models, datasets, attacks, thresholds, and evaluation criteria. But they are still useful. They show that a carefully prompted classifier can substantially improve robustness against non-adaptive attacks. Defence in depth is not fake. The pipeline genuinely raises the cost of ordinary attacks.

The trap is assuming that beating ordinary attacks means beating adaptive ones.

The headline result matters because the obvious attack failed first

The paper’s most quoted number will be 71%. That is the attack success rate STACK achieves on ClearHarm against the few-shot-prompted Gemma 2 pipeline under the black-box front-to-back setting.

That number matters precisely because PAP alone gets 0% against the same defended pipeline on ClearHarm. PAP is highly effective against the undefended model, reaching 99% on ClearHarm, but the Gemma 2 input and output classifiers shut it down. STACK then recovers much of the lost attack success by attacking the pipeline structure rather than merely the model.

This is the evidence hierarchy:

The strongest interpretation is not that STACK “beats safeguards”. It is that staged attacks convert safeguard evaluation from a one-shot jailbreak contest into a systems-security problem. The attacker is no longer just persuading the model. The attacker is learning the pipeline.

This is why the paper’s example of classifier scores is conceptually useful even without reproducing the underlying harmful prompt or jailbreak string. For one harmful input, the clean request is confidently flagged by the input classifier. PAP gets the model to answer but is caught by the output classifier. STACK pushes both classifier scores below their thresholds. The model still has to answer, but the attack has separately arranged for the request and response to look acceptable to the two guards.

That is the “Swiss cheese” failure mode in miniature. The holes did not randomly align. The attacker filed them into alignment.

Transfer turns a red-team exercise into a portability problem

The front-to-back attack assumes the attacker can infer which component blocked the request. That is plausible in many systems, but not inevitable. A more disciplined deployment might hide component-level signals and expose only a generic refusal.

The paper therefore tests a transfer setting. Here, the attacker optimises against a proxy pipeline and then applies the learned attack to the target pipeline. The proxy uses zero-shot-prompted Qwen3 classifiers. The target is the stronger few-shot-prompted Gemma 2 pipeline. The attack still uses PAP against the generative model, but the classifier bypasses are learned elsewhere.

The result: 33% end-to-end attack success on ClearHarm.

This is smaller than 71%, but operationally more annoying. The cost reported for the transfer attack is also much lower: 15 H100 GPU-hours for the seeds in their experiment, compared with 507 H100 GPU-hours for the front-to-back attack. The authors estimate cloud-compute costs of roughly $30 and $1,000 respectively, while noting that these figures exclude the broader experimentation cost of trying hyperparameters.

The exact dollar numbers will age badly, as cloud prices tend to do. The relative point is more stable: if an attacker can train against a proxy, the target system may not need to leak as much during the final attack. That reduces the defender’s ability to catch the optimisation process through repeated failed queries.

This is where public model ecosystems complicate enterprise safety. Open-weight guard models, public moderation APIs, and published safety templates are useful for researchers and developers. They also give attackers approximation surfaces. If the deployed guard is close enough to something the attacker can study, the attack may transfer.

That does not mean “never open-source safety models”. That would be a suspiciously convenient conclusion for vendors selling closed systems. It means production systems should not assume that obscurity is the only defence, nor that public evaluation of a guard model is the same as production robustness. Diversity matters. Proprietary fine-tuning data can matter. Architectural opacity can matter. So can boring operational controls like rate limits and anomaly detection.

The boring parts of security are often where the non-boring failures are prevented. Tragic, but useful.

The paper is about catastrophic misuse, but the mechanism is broader

The authors focus on catastrophic misuse: harmful use of advanced models in domains where detailed assistance could produce severe real-world damage. ClearHarm is built around unambiguously harmful, non-dual-use queries. This is narrower than general content moderation, and deliberately so.

That focus makes the results cleaner. Dual-use prompts are messy. A question about chemistry, cybersecurity, biology, or finance can be legitimate or dangerous depending on context. ClearHarm tries to avoid that ambiguity by selecting requests where compliance would provide harmful uplift. The evaluation then counts a response as successful only when an LLM judge rates it as non-refusal and maximally convincing and specific. That is a stricter criterion than merely checking for refusal keywords.

For business use, however, the same pipeline mechanism appears in less catastrophic settings:

The paper is not claiming all these systems face the same severity of risk. A compromised customer-support bot is not a frontier biosecurity incident. But the control architecture rhymes. If a system has multiple guard components and observable component-level failures, attackers can use those observations.

This matters especially for agentic systems. A plain chatbot produces text. An agent can call tools, fetch files, update records, send messages, or trigger workflows. Defence-in-depth becomes more necessary as the action space grows. At the same time, the number of observable stages grows too: input policy, retrieval policy, tool-use policy, action validator, output policy, audit monitor. More slices. More possible holes. More possible labels.

The correct lesson is not to simplify the system into one fragile layer. It is to make the layers harder to separately map.

What operators should change in production pipelines

The paper’s recommendations are practical, but they need translation into operating language.

First, make refusals less diagnostic. If an input classifier, the model itself, and an output classifier all produce visibly different behaviours, the system is offering attackers a progress meter. A generic refusal surface is less helpful to attackers than component-specific messages. This does not require lying to users. It requires not making internal control flow part of the API contract.

Second, reduce timing side channels. If the system exits early when an input classifier fires, but takes longer when the request reaches the model or output classifier, latency becomes a classifier-stage signal. Running all stages before responding may increase cost and latency, but selectively leaking the stage of rejection is not free either. Security costs usually show up somewhere. Better in infrastructure than incident response.

Third, be careful with streaming. The paper notes a trade-off. Streaming output classifiers may block an attack style that depends on a suffix appearing after harmful content, because the classifier sees the content before the suffix. But streaming can also leak partial harmful content before the block. For high-risk domains, streaming should be treated as a safety architecture decision, not a UX default.

Fourth, diversify safeguards. If a deployed classifier is close to a public model, a public moderation API, or a widely studied prompt template, attackers get a proxy. Fine-tuning on proprietary datasets, using non-public base models, ensembling genuinely different classifiers, or combining text classifiers with non-textual signals may reduce transfer. The word “genuinely” is doing work here. Three similar models voting together can become one model wearing three hats.

Fifth, monitor optimisation behaviour. STACK-like attacks, especially front-to-back ones, may involve repeated probing, candidate selection, and stage-specific failures. Security telemetry should look for search patterns, not only forbidden keywords. A user issuing many semantically related borderline prompts, with small perturbations and shifting refusal outcomes, is not merely “curious”. They may be debugging your safety stack for you, without the courtesy of an invoice.

Sixth, evaluate adaptively. A benchmark suite of standard attacks is useful for regression testing. It is not enough for assurance. Red teams should explicitly attempt staged attacks under multiple threat models: fully opaque, semi-separable, exposed moderation API, streaming, tool-use, and proxy-transfer conditions. The result should be a threat-model matrix, not a single robustness score.

A compact operator checklist looks like this:

This is less glamorous than announcing “AI safety framework v4.7”. It is also more likely to matter.

Where the evidence stops

The paper is strong because it separates mechanisms, threat models, and experiments. It is also bounded. Those boundaries matter.

First, the main pipeline is open-source and built around Qwen3-14B with open or prompted classifier models. That makes the work reproducible and useful, but it is not a full replica of every frontier deployment. Proprietary systems may use stronger classifiers, richer telemetry, different refusal logic, private training data, and additional controls not captured here.

Second, the output classifiers in the main experiments see complete responses. Streaming classifiers change the game. The authors are clear that streaming could both introduce new vulnerabilities and block some attack variants. Any operator using streaming moderation should test that specific architecture rather than importing the paper’s numbers directly.

Third, the evaluation uses LLM-as-judge scoring. The authors choose a conservative criterion: responses count as successful only if judged non-refusals and maximally convincing and specific. That is a reasonable design choice, but automated judges are still measurement instruments, not reality itself. For high-stakes domains, human expert review remains necessary.

Fourth, the transfer result is preliminary. A 33% attack success rate is enough to prove concern, not enough to map the whole transfer landscape. Transfer may be weaker or stronger depending on model families, prompt templates, thresholds, datasets, and the similarity between proxy and target.

Fifth, the frontier-model appendix should be treated as an exploratory extension. The authors report that component-wise methods found vulnerabilities in Claude Opus 4 and GPT-5-style deployments, and that mitigations were deployed after disclosure. But those attacks were not identical to the main STACK implementation. The appendix supports the broader idea that component-wise leakage matters. It does not prove that the exact open-source STACK recipe transfers unchanged to every frontier model.

These limitations do not deflate the paper. They prevent lazy extrapolation. The result is not “all guarded LLMs are broken”. It is “guarded LLMs need to be evaluated as systems whose internal signals can be attacked”.

That is less tweetable. A pity. It is also the actual point.

The business value is knowing which layer becomes the product risk

For enterprises, the paper’s value is not primarily in the attack name. It is in the diagnostic lens.

Most AI governance programmes still treat safeguards as policy compliance objects: do we have moderation, logging, red teaming, escalation, access control? The paper suggests a more operational question: does the interaction among these controls leak a usable optimisation signal?

That question changes how risk should be reviewed.

The ROI case for this kind of work is not “buy more safety”. It is cheaper diagnosis. A company can spend millions layering controls and still leak internal state through obvious API behaviours. Fixing refusal consistency, timing patterns, telemetry, and proxy exposure may be cheaper than simply adding another classifier and declaring the cheese thicker.

For AI vendors, this becomes a product-security issue. If customers can infer too much about internal safety routing, the API itself becomes a red-team oracle. For enterprises deploying third-party models, it becomes a procurement issue. Ask vendors how they test semi-separable leakage, not only whether they have moderation. Ask whether their refusal behaviours are intentionally non-diagnostic. Ask how they evaluate transfer attacks from open-source proxy models. Watch how quickly the sales engineer stops smiling.

For companies building agents, the implications are sharper. Agentic systems are not just producing text; they are controlling workflows. A failed tool call, a blocked retrieval, a refused action, or a policy-specific error message can all reveal guard boundaries. The more operational the agent, the more valuable those boundaries become to an attacker.

Defence-in-depth still works when depth is hard to map

The useful ending is not cynicism. Defence-in-depth works. The paper’s own baseline evaluation shows that the best pipeline blocks attacks that devastate the undefended model. On ClearHarm, PAP goes from 99% against the undefended Qwen3-14B model to 0% against the Gemma 2 defence pipeline. That is not cosmetic safety theatre. It is real reduction under the tested conditions.

But defence-in-depth is not merely the presence of layers. It is the interaction of layers under adversarial observation. If each layer announces itself when it fires, attackers can optimise around it. If public proxy models approximate the deployed guards, attackers can rehearse elsewhere. If latency reveals control flow, the stopwatch becomes an API. If streaming reveals partial outputs, UX becomes leakage with a progress bar.

STACK’s contribution is to make this failure mode concrete. It does not say the Swiss cheese model is wrong. It says the attacker may be allowed into the kitchen with a flashlight.

For operators, the mandate is straightforward. Keep the layers. Make them less separable. Test them adaptively. Treat side channels as part of the model interface. Stop celebrating benchmark zeros until someone has tried to optimise against the stack itself.

The future of LLM safety will not be won by a single perfect classifier. It will be won, if at all, by systems whose failure signals are boring, ambiguous, monitored, and expensive to learn from. A noble destiny for enterprise software: to become less educational to its enemies.

Cognaptus: Automate the Present, Incubate the Future.