Safe on Paper, Lost in the Prompt

TL;DR for operators

A safety-aligned image model can keep its FID and CLIPScore nearly unchanged while becoming materially worse at following ordinary instructions. It may still generate a plausible bird, vase, or product scene, but quietly miss the requested color, quantity, relationship, or attribute.

The paper identifies a mechanism behind this failure. When safety tuning modifies the text encoder, benign prompt embeddings can become compressed and their semantic neighborhoods can be rearranged. Distinctions that the original model represented clearly begin to blur. The authors call this semantic collapse.¹

Their proposed method, Structure-Aware Geometric Regularization, or SAGE, adds two controls during safety alignment:

1. It prevents the overall embedding space from shrinking.
2. It preserves similarity relationships among nearby benign prompts.

On Stable Diffusion v1.4, SAGE reaches a TIFA score of 75.4, close to the unmodified model’s 76.3 and above the DES safety baseline’s 71.6. Its average attack success rate is 1.2%, compared with 67.6% for the unmodified model. It also largely preserves performance on GenEval, long prompts, and additional compositional benchmarks.

For organizations evaluating image-generation systems, the immediate implication is simple: do not accept global image-quality metrics as evidence that safety alignment preserved utility. Require structured tests for counting, color binding, attributes, spatial relationships, and long instructions. When the intervention changes the text encoder, embedding spread and neighborhood preservation may also serve as useful diagnostic signals.

The boundary is equally important. The paper studies Stable Diffusion variants, mostly automated evaluators, and safety methods that edit text representations. It does not prove that semantic collapse explains failures caused by UNet modification, output filtering, proprietary model tuning, or every form of safety intervention.

The dashboard says the model is fine

Imagine a company testing a newly safety-aligned image generator.

The unsafe-content rate falls sharply. FID improves. CLIPScore barely moves. The deployment dashboard is green, the governance slide is reassuring, and someone inevitably declares that the safety–utility trade-off has been “solved.”

Then the model is asked to generate three red containers beside one blue container.

It produces four containers, approximately colorful.

Or it is asked for a horned owl wearing a graduation cap and holding a diploma. It supplies an owl, which is apparently close enough for the metric department.

This is the problem examined in The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models. The paper argues that several widely used utility metrics answer a much easier question than operators assume.

FID asks whether the overall distribution of generated images resembles a reference image distribution. It does not ask whether an individual image obeyed its prompt.

CLIPScore estimates broad image–text similarity. It can recognize that an image and prompt concern roughly the same subject while overlooking whether the image contains the right number of objects, whether attributes are attached to the correct objects, or whether one object is above rather than below another.

Those metrics are not useless. They are simply being promoted beyond their job descriptions.

The result is an evaluation loophole: a safety-aligned model can remain visually plausible and globally related to its prompts while losing the fine semantic distinctions that make it useful in production.

Coarse scores preserve plausibility, not instructions

The paper tests this claim using TIFA, a structured text-to-image evaluation benchmark. Instead of comparing an image and prompt through one broad similarity score, TIFA decomposes the prompt into visual questions.

Does the image contain the requested object?

Is the object the specified color?

Are there three objects rather than two?

Is the small book sitting on the large book?

This changes the evaluation target from “Does the image look generally relevant?” to “Did the model satisfy the instructions?”

That distinction produces a rather less flattering picture of safety alignment.

The unmodified Stable Diffusion v1.4 model scores 76.3 on TIFA. DES, a strong text-encoder safety-alignment baseline, scores 71.6—a reported 6.2% relative decline. STEREO scores 69.9, while Adv-Unlearn falls to 63.1. SafeCLIP and SafeRCLIP fall further, to 60.1 and 60.7.

Yet DES produces a better FID than the base model: 16.23 rather than 17.23, where lower is better. Its CLIPScore declines only from 26.5 to 25.5.

Viewed through the conventional dashboard, DES appears to have retained most of the model’s utility while adding strong safety controls. Viewed through structured prompt compliance, it has lost considerably more.

The deterioration is also uneven. DES loses 13% on the TIFA food category, even though its category-level CLIPScores remain relatively stable. This matters because production failures rarely distribute themselves conveniently across an average. A model that remains adequate for landscapes but becomes unreliable at quantities, materials, or product attributes can pass an aggregate review and still fail its intended workflow.

The misconception to retire is that an unchanged CLIPScore means an unchanged model. It means only that one broad projection of model behavior has not changed very much.

A company would not evaluate a database migration by checking that the average page still contains approximately the same words. Image-model evaluation deserves at least comparable dignity.

Safety tuning compresses distinctions the generator still needs

Finding a hidden performance loss is useful. Finding a plausible mechanism is more useful.

The paper focuses on safety methods that modify the text encoder while keeping the diffusion model’s UNet frozen. The text encoder maps prompts into embeddings that condition image generation. Those embeddings do not merely represent isolated prompts; their arrangement encodes relationships among concepts.

Consider several prompts:

• three golden retrievers;
• three dogs;
• two golden retrievers;
• a cat.

In the original embedding space, the first three should be related but distinguishable. “Three dogs” may sit close to “three golden retrievers,” while “two golden retrievers” differs along quantity. “A cat” should remain more distant.

Safety tuning attempts to move unsafe prompts away from unsafe semantic regions or toward neutral targets. Existing methods often include a utility loss that keeps each benign prompt embedding near its original position. But treating each prompt independently does not guarantee that the space retains its overall structure.

Many individually small movements can still produce a collectively distorted map.

The paper identifies two such changes.

Embedding contraction reduces representational room

The first is a reduction in embedding spread.

For a batch of normalized prompt embeddings, the authors measure average squared distance from the batch mean. They then compare the spread of the safety-aligned encoder with that of the original encoder.

A ratio below one means the adapted embeddings have contracted.

This contraction matters because closely packed representations have less room to preserve distinctions among objects, quantities, attributes, and relationships. The model may still recognize the general subject, but increasingly treat nearby instructions as interchangeable variations.

The generator does not necessarily forget what a dog is. It becomes less precise about which dog instruction it received.

Neighborhood distortion rearranges semantic relationships

Spread alone is not enough. An embedding space could maintain its overall size while still scrambling local relationships.

The authors therefore examine each prompt’s nearest neighbors before and after safety alignment. They calculate how much the adapted model preserves the original top-$K$ neighborhood using Jaccard overlap.

Low overlap means prompts that were previously close are no longer close, or unrelated prompts have entered the local neighborhood.

This is the second part of semantic collapse: the space’s relational logic becomes distorted.

The paper reports that these geometric changes track structured utility loss. DES retains an embedding-spread ratio of 0.80 and a neighborhood Jaccard score of 0.52, with a TIFA score of 71.6. Adv-Unlearn falls to a spread ratio of 0.71 and Jaccard of 0.50, alongside a TIFA score of 63.1.

The authors’ method reaches a spread ratio of 0.96, Jaccard of 0.63, and TIFA of 75.4.

The proposed causal story is therefore:

Text-encoder safety tuning
          ↓
Benign embeddings contract and local neighborhoods shift
          ↓
Fine distinctions among prompts become less stable
          ↓
Counts, attributes and relationships are lost in generation
          ↓
FID and CLIPScore continue to report respectable aggregate utility

The paper shows strong correlations across methods and semantic categories. It does not, however, establish geometry as the sole cause through a fully controlled causal identification strategy. Geometry is a well-supported diagnostic mechanism here, not a universal law of alignment.

That distinction will disappoint only those who require every useful paper to solve philosophy before lunch.

SAGE preserves the map while changing the restricted routes

SAGE adds two geometric regularizers to the DES training framework.

The first, Embedding Spread Preservation (ESP), prevents the adapted encoder’s spread from dropping below the base encoder’s spread.

Importantly, the penalty is directional rather than symmetric. It penalizes contraction but does not force the adapted space to match the original spread exactly. The encoder can still expand or reorganize when required for safety; it is merely discouraged from compressing benign semantics.

The second, Local Structure Alignment (LSA), preserves similarity patterns among nearby prompts. Rather than matching every possible prompt pair, SAGE identifies each prompt’s closest neighbors under the base encoder and encourages the adapted encoder to retain the relative similarity structure within those local groups.

This is more expressive than pointwise preservation.

Pointwise alignment says:

Keep each benign prompt close to where it began.

Local structural alignment says:

Also preserve which prompts are closer to one another and by how much.

The difference resembles preserving the coordinates of individual buildings versus preserving the street network connecting them. A city can move every building only slightly and still produce a remarkably unusable map.

Why SAGE perturbs the local structure objective

Naively preserving the base model’s semantic geometry creates another problem. The original geometry may contain associations with the unsafe concept being removed. Reconstructing it too faithfully can restore some of the unwanted capability.

To avoid this, the final LSA formulation perturbs adapted embeddings along the specified unsafe concept direction before enforcing local consistency. In the main experiments, that direction is associated with nudity.

The aim is to preserve benign relationships under the very displacement introduced by safety alignment, rather than simply reconstructing the unaligned space.

This detail is central to the method’s safety performance. It is not decorative algebra added so the method can have enough equations for publication.

Structured utility returns without a large safety rebound

The main experiment uses Stable Diffusion v1.4. The authors fine-tune the text encoder while freezing the rest of the model, using 6,911 safe–unsafe prompt pairs from the sexual-content category of the CoPro dataset.

The evaluation covers four distinct questions:

TIFA shows near-base structured fidelity

SAGE scores 75.4 on TIFA, compared with 76.3 for the unmodified model.

That remaining gap should not be erased rhetorically. SAGE does not fully restore the base model. It reduces the loss from 4.7 TIFA points under DES to 0.9 points.

Relative to DES, the paper reports a 5% improvement. Relative to STEREO, the improvement is 7.3%.

The category results are particularly informative. DES scores 71.1 on food prompts, compared with 84.1 for the base model. SAGE reaches 83.5. This is precisely the type of category-specific recovery that CLIPScore failed to expose.

SAGE also records a CLIPScore of 26.4, nearly matching the base model’s 26.5, and an FID of 15.93, the best reported among the compared methods.

The FID result is welcome, but it is not the paper’s main intellectual contribution. The point is not that SAGE wins the old scoreboard. The point is that the old scoreboard was insufficient.

Safety remains close to strong baselines

Across MMA-Diffusion, Sneaky Prompt, I2P sexual prompts, Ring-A-Bell and P4D, the unmodified model has an average attack success rate of 67.6%.

SAGE reduces that average to 1.2%.

DES records 1.0%, Adv-Unlearn 0.4%, and STEREO 2.5%. SAGE is therefore not the absolute winner on the average safety column. Instead, it occupies a more attractive part of the joint safety–utility surface.

Adv-Unlearn’s 0.4% average ASR arrives with a TIFA score of 63.1, 17.3% below the base model. DES remains slightly safer by the reported average but loses substantially more structured utility. SAGE accepts a fractional increase in attack success relative to DES in exchange for recovering most of the lost prompt fidelity.

That is a trade-off, not magic. It is simply a much better trade-off than global utility metrics had led the field to measure.

GenEval confirms the result is not peculiar to TIFA

On the selected GenEval tasks, the base model scores 60.8. SAGE scores 59.8, a 1.6% relative decline.

DES scores 56.7, Adv-Unlearn 54.2, and STEREO 46.7. MACE, SafeCLIP and SafeRCLIP suffer much larger compositional losses.

GenEval is a robustness check rather than a second thesis. Its role is to show that the result survives a different structured benchmark focused on object presence, color, counting and two-object composition.

It does.

The authors omit GenEval position and attribute-binding categories because the base model already performs poorly on them. That is a reasonable decision for comparing degradation, although it also means the reported average is not a comprehensive compositional score.

The ablations reveal two controls with different jobs

The component study begins with DES and adds ESP, LSA, or both.

ESP alone improves TIFA by 2.9 points while leaving average ASR close to DES. This supports the claim that preventing global contraction protects utility.

LSA alone produces the highest TIFA result in the ablation, 76.0, slightly below the base model’s 76.3. But its ASR rises to 1.7%. Preserving local semantic relationships recovers more capability, including some capability safety tuning was supposed to suppress.

The combined model scores slightly lower on TIFA than LSA alone, but restores stronger safety and obtains the best CLIPScore of the four configurations. This is why the full system should be read as a balancing mechanism rather than a pure utility maximizer.

The supplementary ablations make that interpretation clearer.

A symmetric spread penalty, which constrains both contraction and expansion, performs worse than the directional ESP formulation on the sampled ablation set: 2.4% average ASR and 25.9 CLIPScore, versus 1.1% and 26.4 for directional ESP. Preventing collapse works better than freezing the entire space in place.

Likewise, standard LSA without unsafe-concept perturbation reaches a slightly higher CLIPScore of 26.5 but an average ASR of 4.3%. Adding the perturbation lowers average ASR to 1.1% with CLIPScore of 26.4.

These are not minor tuning curiosities. They show why “preserve the geometry” is too crude an instruction. SAGE must preserve enough geometry to retain benign distinctions without reinstating the unsafe direction being removed.

The neighborhood-size study selects $K=15$ as the best tested balance. Very small neighborhoods miss useful structure; some larger settings include weaker relationships or destabilize the safety result. Because these experiments use reduced subsets for computational efficiency, the exact value should be treated as a configuration result for this setup, not an industry constant carved into silicon.

The appendix strengthens the mechanism, not the marketing claim

Several supplementary experiments extend the evidence, although they do not all carry equal weight.

Pairwise distances are better preserved

Using 400 benign TIFA prompts, the authors compare pairwise cosine-distance matrices before and after alignment. DES produces visibly greater distortion, while SAGE preserves the base model’s distance ordering more closely.

This supports the geometric diagnosis directly. It does not independently prove that every preserved pairwise relationship is necessary for generation quality, but it aligns with the spread, neighborhood and structured-utility results.

Long prompts expose the operational value

On a sample of 100 DPG-Bench prompts averaging 66.4 words, judged by GPT-4o, the base model scores 56.1. SAGE reaches 53.0, while DES scores 44.5, STEREO 44.7 and Adv-Unlearn 34.0.

This is a small exploratory evaluation, not a definitive long-instruction benchmark. Still, it is operationally revealing. Dense prompts accumulate constraints, giving semantic collapse more opportunities to drop details.

For creative production, advertising layouts, product visualization and design assistance, this is closer to how demanding users actually work than “a photograph of a dog.”

Additional compositional benchmarks show a consistent pattern

On T2I-CompBench++, SAGE matches or exceeds the base model on several reported dimensions, including color, shape and texture, and remains among the strongest safety-aligned methods across spatial and multi-constraint tasks.

Individual benchmark metrics should not be merged casually into a single victory claim. Their scales and evaluators differ. Their value is convergence: TIFA, GenEval, T2I-CompBench++ and the long-prompt test all point in the same direction.

Adaptive attacks do not immediately undo the defense

Under the white-box U3-Attack, tested on 310 prompts with two generations each, SAGE records attack success rates of 1.3%, 2.3% and 2.3% under three different safety classifiers. Adv-Unlearn records 6.8%, 2.6% and 3.2%.

This is stronger safety evidence than transfer-based attacks alone because the adversary adapts to the target. It is still bounded by the attack formulation, classifier coverage and unsafe concepts tested. “Survived one adaptive attack” should never be translated into “robust to adaptive attackers” without qualifiers. Security departments have suffered enough from grammar.

Broader concepts and newer checkpoints are promising but less conclusive

For violence, illegal content, hate, self-harm, harassment and shocking content on the I2P benchmark, SAGE reports an average ASR of 1.4%, compared with 2.2% for DES, while modestly improving CLIPScore.

This suggests that the approach can extend beyond sexual content when supplied with additional prompt pairs and explicit unsafe directions.

On Stable Diffusion v2.1, however, the picture is more restrained. SAGE scores 75.7 on TIFA, versus 79.1 for the base model and 75.9 for AlignGuard. Its safety is substantially stronger than AlignGuard under the tested attacks—12.2% average ASR versus 36.5%—but the utility recovery is not as close to the base model as it is on v1.4.

That is still useful evidence. It is simply evidence of generalization with changed magnitudes, not effortless portability.

Procurement needs a prompt-compliance test, not another beauty score

The immediate business value of this paper is not a new regularizer that every company must implement next Tuesday.

Most companies will not train their own safety-aligned text encoder. They may procure a hosted model, license an enterprise checkpoint, or rely on a platform whose alignment method is not disclosed.

The broader value is a better evaluation design.

What the paper directly shows

For the evaluated Stable Diffusion systems:

• FID and CLIPScore can conceal meaningful losses in structured prompt fidelity.
• Text-encoder safety alignment can contract embedding spread and distort local semantic neighborhoods.
• These geometric changes correlate strongly with reductions in TIFA performance.
• Preserving spread and local structure can recover much of the lost utility while maintaining low attack success rates.

What Cognaptus infers for business use

Organizations should construct safety–utility scorecards around the actual tasks the model performs.

A product-imaging system should be tested for exact quantities, materials, colors, placement and branded attributes.

A marketing system should be tested on long briefs containing multiple subjects, exclusions and layout constraints.

A design assistant should be tested for relationship preservation across iterative prompt revisions.

A content platform should evaluate both direct unsafe prompts and adversarial reformulations while separately measuring benign prompt fidelity.

A practical acceptance framework could look like this:

The return on this evaluation is not merely a higher benchmark score. It is fewer failed generations entering expensive review loops.

A model that produces attractive but specification-inaccurate images can look efficient at inference time while moving costs downstream to designers, marketers, compliance reviewers and customers. Global quality scores are particularly bad at pricing that operational burden because they average away the errors people must manually repair.

What remains uncertain

Embedding geometry is most actionable when the organization controls or can inspect the text encoder. Hosted-model customers may not have access to these representations.

Even with access, spread and neighborhood metrics are diagnostics rather than universal service-level objectives. A ratio near one does not guarantee task success, and a changed geometry is not automatically harmful if the downstream model adapts successfully.

The safer operational conclusion is therefore:

Use geometry to investigate capability loss, but use task-specific output tests to decide whether the model is acceptable.

The method does not cover every safety intervention

The paper’s scope is narrower than the general phrase “image-model safety alignment” may suggest.

First, SAGE assumes that safety tuning modifies the text encoder. Methods that alter the UNet, intervene during denoising, filter prompts, classify outputs, or combine several layers of control may fail for different reasons. SAGE’s regularizers do not directly address those mechanisms.

Second, the method requires explicit unsafe-concept directions. The experiments include nudity and extensions involving concepts such as blood and politics, with additional evaluation across several unsafe categories. Real production policies are less tidy. They include overlapping, contextual and culturally dependent categories that may conflict with one another.

Third, the principal experiments use Stable Diffusion v1.4, with supplementary tests on v1.5 and v2.1. This does not establish the same effect size for larger proprietary generators, newer text encoders, multimodal foundation models or heavily modified enterprise systems.

Fourth, much of the evaluation is automated. TIFA uses Qwen-3-32B as the visual question-answering evaluator; the long-prompt study uses GPT-4o; safety tests rely on classifiers including NudeNet, Q16 and a multi-headed detector. These tools make large-scale structured testing feasible, but they introduce their own errors and blind spots.

Fifth, TIFA generates one image per prompt in the reported protocol. Image generation is stochastic, so production evaluation should test multiple seeds when reliability matters. A model that succeeds once and fails four times is not operationally equivalent to one that succeeds consistently, however similar their one-shot benchmark score may be.

These boundaries do not undermine the central finding. They define where it can be used without turning a useful diagnosis into a universal alignment mythology.

Safety alignment needs a semantic balance sheet

This paper exposes a familiar institutional habit: measuring what is convenient, then quietly redefining success around the available measurements.

FID is convenient. CLIPScore is convenient. Neither was designed to certify that a safety-aligned model retained every operationally important distinction in benign prompts.

Once structured evaluation is introduced, the apparent resolution of the safety–utility trade-off becomes less convincing. Several methods are safe partly because they have damaged the representational machinery required for precise instruction following.

SAGE offers a technically coherent response. It does not simply keep each benign embedding near its original point. It preserves the space’s capacity to separate prompts and the local relationships that organize them. Its experiments show that this can recover most of the lost structured utility without reopening the unsafe behavior under the tested attacks.

The business lesson is broader than SAGE itself.

Safety evaluation and utility evaluation cannot remain separate columns assembled from unrelated aggregate metrics. Alignment changes the internal representation used for ordinary tasks. A model can become safer by becoming less semantically competent, and the standard quality dashboard may applaud throughout.

Operators therefore need a semantic balance sheet: what distinctions were removed, what benign distinctions survived, which workflows degraded, and whether the safety improvement remains robust when users stop phrasing attacks politely.

Otherwise, “high utility” may mean only that the model still produces attractive images of approximately what was requested.

Which is an impressive standard for wallpaper. It is a rather weaker standard for a production system.

Cognaptus: Automate the Present, Incubate the Future.