Don’t Let the Oracle Score Your Router
TL;DR for operators A hierarchical selector makes two decisions: first choose a family of models or solvers, then choose a concrete member of that family. If evaluation scores the first decision by quietly granting it the best possible member of the chosen family, the benchmark is measuring a system that cannot actually be deployed. ...