Safety-Alignment

TL;DR for operators A safety-aligned image model can keep its FID and CLIPScore nearly unchanged while becoming materially worse at following ordinary instructions. It may still generate a plausible bird, vase, or product scene, but quietly miss the requested color, quantity, relationship, or attribute. The paper identifies a mechanism behind this failure. When safety tuning modifies the text encoder, benign prompt embeddings can become compressed and their semantic neighborhoods can be rearranged. Distinctions that the original model represented clearly begin to blur. The authors call this semantic collapse.1 ...

The dangerous part is often clearer after the model starts answering Moderation usually begins with the user’s prompt. That sounds sensible. Read the request, classify the risk, block the bad thing, let the good thing through. A tidy little border checkpoint, complete with imaginary clipboard. The problem is that jailbreaks are not polite enough to declare themselves at the border. ...