Architecture Search Has a Training Problem
TL;DR for operators Neural architecture search is not simply a process for selecting layers, connections, and activation functions. It is a process for comparing trained architecture–weight pairs. That distinction matters because an architecture evaluated with poorly optimized weights may look worse than an inferior architecture whose weights happened to train faster. A search system can therefore become very efficient at ranking the wrong candidates. ...