Layers Are Not Independent: Red-Team the Whole AI Safety Stack

TL;DR for operators

A production AI request may pass through preprocessing, a safety guardrail, and a generator, with each stage intended to reduce risk. Cascade¹ shows why evaluating those defenses separately can miss an important failure mode: an exploit at one layer can remove a condition that another defense depends on.

In the paper’s guardrail experiment, random attention perturbation evades the guardrail on 94% of evaluated malicious prompts, versus 82% for targeted token bitflips and 72% for targeted attention bitflips. That 94% figure is not an end-to-end compromise rate. After combining guardrail evasion with a reported 82% generator-jailbreak rate and externally sourced hardware bitflip probabilities, the corresponding calculated full-chain attack-success rate is 0.750 or 0.765.

For teams running compound AI systems, the relevant change is scope. Red teaming should test not only prompts and model behavior but also whether preprocessing services can be disabled, guardrail state can be corrupted, dependencies introduce exploitable software paths, and infrastructure assumptions such as workload isolation actually hold. The evidence does not establish that these chains are feasible in every deployment: the detailed case study assumes knowledge of the pipeline and colocated workloads, and part of its end-to-end probability calculation comes from prior hardware-security research rather than a fully executed physical attack.

A guardrail is only as reliable as the path that executes it

Consider a familiar inference path. An incoming request is modified or normalized by a preprocessing service, checked by a guardrail, and then passed to a generator. Testing each component independently can make this architecture look like defense in depth: a request that escapes one control should still encounter another.

The paper’s case study attacks that assumption rather than merely attacking the generator again.

Its staged path first disrupts a query enhancer through a software vulnerability capable of denial of service. It then perturbs the guardrail’s execution through memory or attention-state fault injection. The generator still receives the request and is attacked using an adversarial suffix. Different stages therefore attack different conditions required for the overall safety pipeline to function.

The key mechanism is dependency between failures. A software exploit does not need to produce the final harmful output itself. It can instead remove preprocessing that complicates a later attack. A memory fault does not have to compromise the generator directly. It can alter what the guardrail attends to while leaving the generator’s copy of the request unchanged. The algorithmic jailbreak then operates under a different set of constraints.

That makes the software and hardware executing a safety control part of the control’s security boundary.

Cascade searches for feasible attack chains rather than isolated exploits

The paper calls the individual attack primitives attack gadgets. Cascade organizes them across algorithmic vulnerabilities, software CVEs, and hardware attacks, then searches for combinations appropriate to a particular deployment.

The repository explicitly includes 100 papers on algorithmic vulnerabilities and 100 software CVEs, together with hardware direct-attack and side-channel techniques whose exact count is not reported. The framework classifies targets using five security properties—confidentiality, integrity, safety, availability, and authorization—and three attacker capability classes covering remote, privileged, and hardware-oriented attackers.

This taxonomy supports a search process rather than serving only as a catalog. Cascade conditions candidate selection on the attacker’s objective, available capability, and the structure of the deployed pipeline. Failed candidates are fed back into the reasoning process so alternative gadgets can be selected until an attack succeeds or the testing budget expires.

The broader examples matter because they keep the paper from reducing to one unusual fault-injection scenario. The source package describes SQL injection extending a RAG-poisoning path, malicious packages enabling query exfiltration, and I/O snooping relaxing a conventional requirement of membership-inference attacks. Cross-stack composition here means any case where a vulnerability in one part of the system changes the prerequisites of an attack elsewhere.

For architecture review, this changes the unit of analysis. The relevant object is not simply “Can component X be exploited?” It is whether failure of component X makes an attack on component Y materially easier or newly feasible.

The 94% guardrail result measures one stage, not the whole compromise

The comparative guardrail experiment is the paper’s clearest quantitative evidence for the mechanism.

Random attention perturbation exceeds targeted token corruption by 12 percentage points and targeted attention corruption by 22 points on guardrail evasion. The result is notable because more precise targeting is not automatically more effective in this evaluated configuration. A comparatively less targeted alteration of attention state can disrupt the guardrail more strongly.

The table also defines the boundary around that result. The full-chain ASRs are calculated using the measured guardrail-evasion rates, an 82% jailbreak rate used in Table I, and representative bitflip probabilities of 97.3% and 99.3% taken from prior Phoenix and Rowhammer-Trojan work. They are therefore probabilistic compositions, not measurements from a single physical end-to-end compromise executed entirely inside the authors’ testbed.

The source package separately reports an LLMart adversarial-suffix experiment with successful suffix generation for 41 queries and an overall generator jailbreak success rate of 80%. Optimization was allowed up to 1,000 GCG iterations and averaged 123 minutes per prompt on four NVIDIA L40S GPUs. Suffixes with cross-entropy loss below 1 are reported as highly reliable, whereas those above 7 have very low attack success. These measurements support the generator stage; they do not remove the hardware-feasibility assumptions embedded in the complete chain.

AI safety reviews should include control integrity and infrastructure assumptions

What the paper directly shows. Within its evaluated pipeline, vulnerabilities outside the generator can interfere with controls that normally constrain harmful requests. It also shows that three fault variants behave materially differently and that attack composition can span software, hardware, and algorithmic layers.

What Cognaptus infers for operators. Security teams responsible for compound AI products should treat preprocessing services, guardrail execution, dependencies, memory integrity, and workload isolation as part of safety assurance. This is particularly relevant to cloud and multi-tenant deployments: if attacker colocation is plausible, infrastructure choices can change the feasibility of attacks that model-only evaluations never exercise.

The decision affected is therefore the scope of red teaming. A team deciding whether a release is adequately protected should test whether failures in surrounding services alter the assumptions under which prompt filters, guardrails, RAG defenses, or generator alignment were validated. An architecture review should also identify components whose loss of availability or integrity removes prerequisites for another defense.

This does not imply that guardrails are ineffective. The paper reports that the evaluated guardrail blocks 63% of harmful queries that the generator itself fails to stop. The security concern is precisely that a control with measurable protective value becomes a high-value integrity target.

The deployment boundary is substantial

The detailed proof of concept assumes an attacker who knows the pipeline architecture and can colocate workloads with the target services on the same physical infrastructure. The attacker does not require host-administrator privileges or physical hardware access, but the feasibility of memory-level attacks under those conditions is partly inherited from prior work rather than reproduced as a complete physical compromise here.

Targeted token and attention faults also require knowledge of relevant positions and precise injection. The paper notes that these variants become harder when malicious meaning is distributed across multiple trigger tokens. A proposed route for simplifying such prompts through an offline query enhancer remains a hypothesis rather than an experimentally established result.

The generator attack introduces another condition: optimizing the adversarial suffix requires model-parameter access. Transfer from public to proprietary models is motivated through prior evidence, not demonstrated here for every deployment.

These boundaries keep the evidence at the level of a strong system-security demonstration rather than a universal estimate of real-world compromise probability. They do not weaken the architectural point: once safety depends on several interacting services and execution layers, assurance based only on the model leaves part of the safety boundary untested.

Red teaming has to follow the dependencies

Cascade’s contribution is less about adding another jailbreak technique than about changing how attacks are assembled and searched.

Compound AI systems inherit vulnerabilities from models, application software, dependencies, orchestration, memory, and hardware. Those weaknesses need not cause the same failure independently. Their significance comes from how one changes the conditions under which another attack operates.

For security teams, that makes dependency analysis a necessary complement to component testing. A guardrail can work well under normal execution and still fail to provide the expected protection when its inputs, memory state, upstream services, or infrastructure assumptions are no longer trustworthy.

The paper does not establish how often such chains will succeed in production. It does provide a concrete reason not to assume that multiple safety layers fail independently. For compound AI, the execution stack is part of the safety system.

Cognaptus: Automate the Present, Incubate the Future.