Safe on Paper, Lost in the Prompt
TL;DR for operators A safety-aligned image model can keep its FID and CLIPScore nearly unchanged while becoming materially worse at following ordinary instructions. It may still generate a plausible bird, vase, or product scene, but quietly miss the requested color, quantity, relationship, or attribute. The paper identifies a mechanism behind this failure. When safety tuning modifies the text encoder, benign prompt embeddings can become compressed and their semantic neighborhoods can be rearranged. Distinctions that the original model represented clearly begin to blur. The authors call this semantic collapse.1 ...