The Rare Cases the Classifier Never Asks About

TL;DR for operators

A classifier trained on keyword-selected examples may become increasingly effective at finding familiar cases while continuing to miss entire categories of relevant documents. Asking it to label examples about which it is uncertain does not necessarily correct the problem.

In a rare hate-speech classification experiment, discriminative active learning (DAL) recovered 55% of positive test examples from initially omitted subgroups, compared with 2% for least-confidence sampling, using the same 1,100-document final label budget. DAL also substantially outperformed random sampling on this measure.

For operators building document discovery, compliance monitoring, or complaint intelligence systems, the implication is to measure discovery of previously unrepresented cases alongside aggregate F1, precision, and recall. DAL deserves consideration when initial labels are potentially biased, but it is not a universal winner. Every evaluated classifier on the harder hate-speech task had precision below 0.2.

A growing labelled set can preserve the original omission

Consider a company identifying rare compliance violations in a large document archive. Analysts begin with familiar search terms, label the matching documents, and train a classifier. The system then chooses additional documents for annotation.

This process appears to reduce annotation effort. Yet the original search terms may capture only recognizable descriptions of violations. Documents expressing the same underlying problem differently can remain absent from the labelled set.

An active learning system can select documents where its current classifier has the least confidence. This is least-confidence sampling: query the cases for which the model is least certain about its predicted label.

The difficulty is that the classifier’s uncertainty reflects what it has learned. If it confidently assigns unfamiliar positive cases to the negative class, those cases may receive little attention during subsequent acquisition rounds.

Emily Silcock and Simon Löwe investigate precisely this failure mode in Active Learning for Finding a Needle in a Haystack.¹ Their central contribution is an empirical comparison of how well active learning policies recover positive subgroups entirely absent from the initial training examples.

The researchers use two English-language datasets: AG News, with constructed rare news-topic classification tasks, and Measuring Hate Speech, which identifies comments targeting religious groups. Positive-class prevalence is approximately 1% in both settings.

Every experiment begins with 50 labelled positive and 50 labelled negative documents. Each policy acquires another 1,000 labels over ten rounds, using the same DistilRoBERTa architecture and training budget. Nine policies, including random sampling, are compared across 25 paired runs per configuration.

In the biased Measuring Hate Speech setting, all initial positive examples concern hate speech targeting Muslims. Seven other religion-target subgroups are excluded from the initial labels, although they remain part of the classification target.

The strategy seeking uncertainty recovered almost none of the missing cases

The researchers measure omitted-subgroup recall: among genuinely positive test documents belonging to subgroups absent from the initial labelled data, what fraction does the final classifier correctly identify?

They also measure recall on the initially represented subgroup and overall positive-class F1. This combination matters because a classifier could raise subgroup recall simply by predicting the positive class more frequently, producing additional false positives.

The main comparison is striking.

Source: Section 6.2, Table 2; Appendix D, Table 6. Biased Measuring Hate Speech setting; final means across 25 runs.

Least-confidence sampling recovered only 2% of the omitted-subgroup positives. Random sampling recovered 17%. DAL recovered 55%, with omitted-subgroup recall significantly higher than every competing strategy under the reported Holm-corrected paired comparisons.

DAL’s recall was also relatively balanced between omitted and represented subgroups: 0.55 versus 0.53. Random sampling showed a much larger difference, at 0.17 versus 0.49.

Class-balanced DAL v1 achieved the highest F1 of 0.18, compared with DAL’s 0.16. The latter was not significantly worse under the reported statistical test, although nonsignificance does not establish equivalent performance.

These are different measures of success. F1 captures a balance between aggregate precision and recall; omitted-subgroup recall asks whether the classifier recognizes positive categories it was never initially shown.

Why the discriminator found documents the classifier overlooked

DAL approaches acquisition through a different question.

Instead of asking which documents have uncertain target-class predictions, it trains an auxiliary discriminator to distinguish representations of already-labelled documents from those remaining unlabelled. Documents that appear insufficiently represented in the labelled set become acquisition candidates.

This does not require the target classifier to recognize them as likely positives.

The paper’s acquisition-composition analysis provides a mechanism-relevant explanation. On biased Measuring Hate Speech, DAL selected 101 positive documents across the ten acquisition rounds, including 94 belonging to previously unrepresented subgroups.

Those omitted subgroups constituted approximately 64% of the positive documents available in the pool, yet accounted for approximately 93% of DAL’s acquired positives.

Least-confidence sampling acquired 69 positive documents, with only 39% belonging to initially omitted subgroups.

DAL therefore differed in both the number of positives acquired and which portions of the positive class those acquisitions represented.

This pattern is consistent with the proposed mechanism: uncertainty-based selection can concentrate on what the classifier already recognizes, while discriminator-based selection can reach underrepresented regions of the document distribution.

However, representation-based novelty alone is not sufficient. Core-Set, another acquisition strategy that does not depend directly on target-class probabilities, achieved only 0.16 omitted-subgroup recall on the same task.

The acquisition analysis helps explain DAL’s advantage, but it does not isolate a causal contribution from every component of the method.

The ranking reverses when the classification task changes

The strong DAL result does not carry over as a general ranking of acquisition strategies.

On biased AG News, least-confidence sampling achieved F1 of 0.73, compared with DAL’s 0.63. Both recovered the omitted Sports subgroup effectively, with recall of 0.90 and 0.91, respectively. Nearly all tested active learning strategies achieved omitted-subgroup recall between 0.86 and 0.91.

The failure seen in Measuring Hate Speech was therefore not inevitable whenever initial positive examples excluded a subgroup.

The class-balanced DAL variants reveal a related trade-off. On biased Measuring Hate Speech, class-balanced DAL v1 increased F1 relative to DAL while reducing omitted-subgroup recall from 0.55 to 0.37. Version 2 reduced omitted-subgroup recall to 0.12, below random sampling.

These exploratory modifications use predicted class membership in acquisition balancing. When those predictions are unreliable, balancing can reintroduce dependence on the classifier’s existing mistakes.

The operational objective consequently changes the preferred policy. A team trying to improve classification on recognizable cases may select differently from a team prioritizing discovery of unfamiliar positive categories.

A coverage test belongs in the annotation workflow

For Cognaptus, the practical inference concerns how organizations evaluate annotation strategies before relying on their outputs.

An enterprise document discovery team starting with keyword-selected examples could compare random, uncertainty-based, and discriminator-based acquisition under identical label budgets. Where known positive categories can be identified, some can be deliberately withheld from initialization to test recovery.

The evaluation should track final F1 and precision, recall on represented and omitted categories, and the composition of newly acquired positives. A higher positive-document yield does not necessarily imply broader coverage.

For compliance monitoring or customer complaint classification, this approach can reveal whether an annotation policy reaches unfamiliar descriptions of the events being monitored. It is a proposed operational test, not a business outcome demonstrated by the paper.

Real workflows introduce another difficulty: the missing categories may not be known in advance. Without reference annotations, direct measurement of omitted-subgroup recall may be impossible. Coverage claims would then need to remain narrower than those obtained in the study’s fully annotated benchmarks.

Better discovery is not deployment reliability

Three boundaries materially constrain interpretation.

First, the comparison uses two English-language datasets, one classifier architecture, and a fixed annotation budget. The initial-sample bias is deliberately extreme. Moreover, AG News changes its positive-class definition between biased and unbiased settings, so those configurations cannot isolate initialization effects alone.

Second, subgroup estimates can be unstable when test groups are very small. One Measuring Hate Speech subgroup contains a single test comment. Training epochs were also selected using evaluation test sets, creating a model-selection limitation.

Third, even DAL’s improved coverage did not produce a highly precise hate-speech classifier. Its positive-class precision was approximately 0.09 in the biased setting, and every evaluated strategy remained below 0.2. These systems cannot be treated as reliable automated moderation solely because one recovered more omitted cases.

The study did not test downstream business decisions or estimate how subgroup omissions affect real-world prevalence measurements.

Its strongest conclusion is narrower and more actionable: an annotation strategy can improve a classifier without adequately discovering the positive categories missing from its initial examples.

For organizations using active learning to identify rare events, evidence of progress should include what new categories the system has learned to recognize, not only how its aggregate score has changed.

Cognaptus: Automate the Present, Incubate the Future.