Safety Has a Memory: Why Multimodal Jailbreak Testing Must Follow the Conversation

TL;DR for operators

Safety testing for a multimodal assistant should cover sequences of interactions, not only whether the system refuses one obviously prohibited prompt. In the tested setup, a staged three-turn attack reached a 91.50% attack success rate on LLaVA-7B and 77.31% on GPT-4o, above the three single-turn attack baselines reported for those models.¹ The result does not establish universal failure rates, but it does show that a prompt-level pass can miss vulnerabilities that emerge after earlier turns establish conversational context.

The paper identifies a second design problem: harmful material may occupy only part of a longer answer. Its defense, FragGuard, splits generated text into fragments, asks multiple LLMs to score each fragment, and uses the most severe score as the response-level decision signal. On the paper’s primary comparison, that reduced turn-3 attack success from 91.50% to 0.96% for LLaVA-7B and from 77.31% to 8.08% for GPT-4o. Operators should treat both conversation trajectory and moderation granularity as variables to test, while validating latency, false refusals, judge calibration, fragment sizing, and benign-task performance on their own traffic.

A refusal can be correct and the safety test still incomplete

Consider a routine pre-release check. A product team presents a multimodal assistant with a prohibited request. The assistant refuses. The interaction is recorded as a pass.

That test establishes something narrow: under that prompt and that conversational state, the system did not produce the prohibited response. It does not establish that the same final request will be handled identically after earlier benign exchanges have shaped the context.

Das and colleagues test precisely this gap. Their attack begins with a benign request about an image, moves through an intermediate contextual turn, and only later asks for content associated with harmful text embedded typographically in the image. Once the path of the interaction matters, safety becomes a property of the conversation trajectory rather than only the final prompt.

The third turn changes the measured risk

The main evidence is the direct attack comparison for LLaVA-7B and GPT-4o. The experiment counts an attack as successful when at least one of the primary toxicity judges scores the generated response at or above the paper’s harmfulness threshold.

This comparison is the paper’s main evidence that staged interaction can reveal vulnerability missed by the evaluated single-turn attacks. It should not be read as a universal ranking of jailbreak methods: some baseline values come from prior studies, while other conditions were reproduced or adapted to the same dataset.

The broader five-model experiment serves a different purpose. Using Mistral-7B and Gemini-2.5-Flash as alternative toxicity judges, turn-3 attack success reaches 91.35% for LLaVA-7B, 52.31% for Qwen-7B, 67.50% for GPT-4o, and 68.09% for Gemini-2.0-Flash. Those are all higher than their turn-2 values.

LLaVA-13B is the useful exception: its attack success falls from 24.42% at turn 2 to 16.54% at turn 3. The evidence therefore supports “later turns often expose more vulnerability,” not “every additional turn monotonically weakens every model.”

The authors attribute the pattern to accumulated context and cooperative continuation: an initially harmless exchange may make a later request appear semantically continuous with what the model is already doing. That explanation is plausible within the reported results, but it is a proposed mechanism rather than one isolated through a dedicated causal intervention.

Conversation history should enter the red-team unit of analysis

For product risk teams, the change is concrete. A regression case should no longer be defined only as an adversarial prompt plus an expected refusal. For applications that sustain multimodal conversations, the test unit should also include the preceding interaction path.

That means preserving state across turns, varying when harmful intent becomes explicit, and checking whether an apparently safe early interaction changes later behavior. A model can therefore pass a prompt-level test while failing a conversation-level one.

This does not require treating every long conversation as adversarial. It means adding staged intent escalation to the safety cases most exposed to persistent context: multimodal assistants, agents with session memory, and interfaces where users can iteratively refine image-grounded requests.

Whole-response moderation is also a design choice

Once a system generates an unsafe answer, a separate question appears: how should an output safety layer inspect it?

A whole-response classifier gives one judgment for the entire answer. FragGuard instead divides the response into fixed 400-token fragments and sends each fragment to three defender LLMs. It then takes the maximum toxicity value across every fragment and every judge:

The design is deliberately conservative. One strongly harmful segment is enough to determine the result even if the surrounding answer appears benign.

The primary comparison shows why that granularity matters:

Average toxicity falls as well: from 4.64 to 1.06 for LLaVA-7B and from 3.82 to 1.12 for GPT-4o.

The authors interpret the improvement as evidence that harmful content can be locally concentrated and partially obscured when a long response receives one aggregate judgment. The benchmark results are consistent with that explanation, although the paper does not vary fragment length systematically enough to establish 400 tokens as an optimal unit.

FragGuard is also an output-side control. It does not demonstrate that the target model has stopped generating harmful material internally; it detects generated text and suppresses the response afterward. That distinction matters when comparing base-model alignment with deployment-layer safety.

The extra evaluations strengthen specific claims, not every claim

Two additional tests reduce dependence on the primary evaluator configuration.

First, the paper repeats evaluation with an alternative Mistral-7B plus Gemini-2.5-Flash judge pair. Under that setup, FragGuard’s turn-3 attack success rates are 1.15%, 4.42%, 7.31%, 0.00%, and 0.58% across the five target models. This is primarily a robustness check on whether the defense result survives a change in evaluation judges.

Second, the authors manually score 65 benchmark instances—five from each of 13 categories. For the defended LLaVA-7B and GPT-4o samples, manual ASR and toxicity values exactly match the reported LLM-based evaluation. Attack-side scores differ modestly. This provides useful validation of the automated scoring pipeline, but 65 examples are too few to establish general LLM-judge reliability.

Neither check resolves benign utility. The paper supplies one benign visual-question-answering example in which FragGuard leaves the answer unchanged. That is illustrative evidence, not a utility benchmark.

The deployment hypothesis is stronger than the deployment evidence

The experiments use 520 balanced samples selected from MM-SafetyBench: 40 from each of 13 prohibited categories. Five target vision-language models are evaluated, and the main results are supported by alternative judges and limited manual review.

That is substantial benchmark evidence for two propositions: staged conversations deserve explicit adversarial testing, and fragment-level screening can outperform whole-response screening under the tested conditions.

Several deployment questions remain open. The paper does not report sensitivity analysis over fragment sizes or toxicity thresholds. Its defense depends on external LLM judges whose calibration and correlated errors become part of the safety architecture. Higher refusal rates can improve adversarial blocking while also creating unacceptable false refusals on legitimate traffic. Additional moderation calls also introduce inference cost and latency.

There are no reported confidence intervals or formal significance tests for the headline comparisons, no broad benign-task evaluation, and no deployment-scale adversarial study. The authors themselves caution against translating the benchmark numbers directly into real-world safety guarantees.

For operators, the paper therefore supplies a testing and architecture hypothesis rather than a finished policy: evaluate the interaction path, inspect output at more than one granularity, and measure the resulting trade-offs on the application that will actually ship.

A clean refusal remains useful evidence. It is simply evidence about one point in a much larger conversational state space.

Cognaptus: Automate the Present, Incubate the Future.