The checklist problem that exposes the model Checklist tasks look boring. That is exactly why they are useful. Ask an LLM to write a formal email under 50 words, include one required term, avoid another term, and return the result as JSON. None of this sounds intellectually difficult. No theorem proving. No multimodal reasoning. No dramatic benchmark leaderboard screenshot. Just instructions. ...
Failure logs are usually treated as evidence for the prosecution. A model is asked to produce a concise compliance summary with three bullet points, mention two risks, avoid prohibited claims, and end with a recommendation. It produces three bullets, correctly identifies the risks, avoids the prohibited claims—and forgets the recommendation. Under a strict binary reward, the response receives a zero. Under a partial-credit reward, it might receive 0.75. The first signal says nothing useful happened. The second says something useful happened, but not precisely what. ...