The Rare Cases the Classifier Never Asks About
TL;DR for operators A classifier trained on keyword-selected examples may become increasingly effective at finding familiar cases while continuing to miss entire categories of relevant documents. Asking it to label examples about which it is uncertain does not necessarily correct the problem. In a rare hate-speech classification experiment, discriminative active learning (DAL) recovered 55% of positive test examples from initially omitted subgroups, compared with 2% for least-confidence sampling, using the same 1,100-document final label budget. DAL also substantially outperformed random sampling on this measure. ...