Safety Benchmarks

TL;DR for operators A moderation team rarely needs a model that is merely “best at safety.” It needs a model that catches the relevant harms at the point where moderation occurs, without creating unacceptable false positives or latency. A large benchmark by Afshin Orojlooyjadid and Hitesh Patel compares 53 specialized moderators and general-purpose language models across 11 public safety datasets.1 Its strongest operational finding is not a new overall winner. It is that the ranking changes with the safety problem and with what the moderator is allowed to inspect. ...

TL;DR for operators A production agent can refuse a dangerous request after its tools have already changed a repository, sent a message, or altered an account. That is why the final response alone cannot establish whether the system behaved safely: stated refusal, attempted action, and persistent environmental change may point to different conclusions. ...