TL;DR for operators
The classifiers have been tested, the predictions are fixed, and the evaluation meeting expects the metric table to reveal an obvious winner. Yet on the yeast multilabel dataset, one way of averaging F1 ranked BR-kNN first and BR-SVM last, while another produced the exact opposite ordering. Nothing about the models or predictions changed; only the unit given equal influence changed.
That distinction is an acceptance-policy choice, not a reporting preference. Pooling all label decisions gives more influence to units contributing larger denominators; macro averaging instead asks whether every class should count equally regardless of prevalence; exemplar averaging judges success separately for each complete case. Evaluation owners should choose the rule that matches where operational cost falls, then test whether the preferred model remains the winner under other defensible rules. Otherwise, a team can approve the wrong model, vendor, or deployment configuration simply because its averaging convention rewards a different unit from the one the operation is meant to protect.
One metric table, two opposite winners
A model-evaluation meeting usually begins after the expensive work is complete. Several classifiers have been trained, the test predictions are available, and the metric table is expected to identify the candidate for deployment. The averaging selector beside F1 can look like formatting: micro, macro, weighted, or samples.
The paper’s real-data example shows that this selector can determine the winner. Four multilabel classifiers were evaluated on the same yeast predictions. Under macro-F1, BR-kNN scored 0.391 and ranked first, while BR-SVM scored 0.356 and ranked last. Under exemplar-F1, BR-SVM scored 0.624 and ranked first, while BR-kNN scored 0.598 and ranked last. The Kendall rank correlation between the two orderings was $-1$, the maximum possible disagreement.
Nothing about training changed between those rankings. Macro-F1 gave each of the 14 labels equal influence, including a rare label with prevalence 0.014 and a common label with prevalence 0.751. Exemplar-F1 instead gave each of the 2,417 examples equal influence and judged the completeness of its predicted label set. BR-kNN looked strongest under equal treatment of labels; BR-SVM looked strongest under equal treatment of cases.
For a procurement team comparing vendors, a governance committee approving a model, or a product owner selecting a deployment configuration, the risk is specific: the chosen averaging rule may reward a unit different from the one carrying operational cost.
The average decides what receives equal standing
The framework begins with two matrices: actual labels and predicted labels. Their rows represent examples; their columns represent classes or labels. Element-wise comparisons produce matrices for true positives, false positives, false negatives, and true negatives. Three summation directions then generate the familiar evaluation families.
| Aggregation | What receives influence | Operational question it answers |
|---|---|---|
| Global, or micro | Individual label decisions are pooled; units with larger measure denominators contribute more | How well did the system perform across all evaluated decisions? |
| Column-wise, then macro | Every class or label receives equal weight | How well does the system serve the average class, regardless of prevalence? |
| Column-wise, then weighted | Per-class results are weighted, commonly by support in library implementations | How well does the system perform after restoring prevalence-based influence to class results? |
| Row-wise, or exemplar | Every complete example receives equal weight | How well does the system handle the average customer, document, image, or case as a whole? |
This gives the paper its main generative result. Any binary measure expressible through $TP$, $FP$, $FN$, and $TN$ can be extended by substituting global, column-wise, or row-wise counts. Precision, recall, F1, Jaccard, balanced accuracy, G-mean, and MCC do not require separate derivations for each multiclass or multilabel variant. The aggregation operator supplies the extension.
The same representation also exposes misleading dashboard abundance. In ordinary multiclass classification with one predicted class per example, micro-precision, micro-recall, and micro-F1 all equal accuracy. Micro-Jaccard is a monotonic transformation of accuracy. Displaying all of them may create the appearance of corroboration while adding no independent model-selection information.
Micro scores contain a weighting policy
The central algebra is compact. For a ratio measure $M_j = \mathrm{num}_j / \mathrm{den}_j$ calculated locally for class or example $j$,
A micro score is therefore a weighted average of local scores, with weights determined by the measure’s denominator. For recall, the denominator is the number of actual positives, so larger classes exert more influence. For precision, the denominator is the number of predicted positives, so influence also depends on the classifier’s output pattern. For F1, the denominator combines actual positives, predicted positives, and errors.
This corrects a common simplification. Micro averaging does not merely “handle imbalance better,” and macro averaging does not merely “focus on minority classes.” Each embeds a particular allocation of influence. The paper expresses their difference as
under its normalized weighting definition. When high-weight classes also receive high local scores, micro exceeds macro. When a model performs better on low-weight classes, the sign can reverse. The gap is thus diagnostic evidence about where the classifier performs well relative to the units that dominate the denominator.
Aggregation also changes the loss being approximated. The paper links micro evaluation to per-label loss, macro evaluation to class-reweighted loss, and exemplar evaluation to per-instance loss in multilabel settings. A team optimizing for macro performance may need reweighted training or class-specific threshold tuning; selecting by macro after training only for pooled accuracy can create objective mismatch.
The experiments verify mechanisms before testing selection risk
The empirical section contains four studies with different evidential roles. Treating all four as equivalent benchmarks would overstate the evidence.
| Study | Likely purpose | What it supports | What it does not establish |
|---|---|---|---|
| Four-class skew simulation | Theoretical verification and sensitivity test | With conditional classifier behavior fixed, changing prevalence moves micro scores and can leave macro recall nearly flat | General performance across deployed datasets |
| 1,200 random five-class classifiers | Direct verification of the covariance identity | The observed micro–macro gap follows the derived weighting relationship; typical F1 gaps in the simulation were narrower than recall gaps | A universal bound on practical ranking changes |
| Soft-label triangular-norm experiment | Sensitivity test for an extended setting | Different rules for combining uncertain truth and prediction alter fuzzy confusion counts; ratio metrics need not preserve raw-count ordering | Probability calibration quality |
| Yeast multilabel comparison | Main practical demonstration | Aggregation alone can reverse classifier selection on fixed predictions | Frequency of such reversals across domains, datasets, and model families |
Figure 1 varies the majority-class prevalence from 0.25 to 0.70 while holding conditional rates constant. Macro recall stays flat while micro recall declines, isolating aggregation as the cause. Figure 2 places all 1,200 simulated classifiers on the predicted covariance relationship to numerical precision. These are strong checks of the algebra because they remove sampling ambiguity, but they are constructed confirmations rather than external validation.
The yeast result carries the business-facing burden. Its complete rank reversal demonstrates a plausible approval failure, yet one benchmark cannot estimate how often procurement or production decisions will change. The correct operational response is a stability test on the organization’s own candidates and data, not a presumption that exemplar or macro averaging will always be preferable.
Choose the operational unit before opening the dashboard
What the paper directly shows: aggregation choices are algebraically related, their divergence can be explained through weights, and they can produce incompatible classifier rankings. It also shows that some commonly reported metric combinations are redundant.
Cognaptus inference: an evaluation specification should treat aggregation as part of the acceptance policy. The owner of the decision should name the unit whose failure is costly, document the chosen aggregation, and report ranking sensitivity across alternatives that correspond to credible operating objectives.
For a multilabel customer-routing system, exemplar measures fit a requirement that each customer receive a sufficiently complete set of destinations. For a safety taxonomy where every hazard class must receive equal oversight despite prevalence, macro measures fit the governance objective. For a high-volume moderation queue where each label decision has similar cost, micro measures may align more closely with throughput-level loss. These are conditional choices; domain cost matrices, thresholds, and minimum per-class constraints may still override a single aggregate score.
A practical approval record should contain four items: the decision-relevant unit, the primary measure and aggregation, the per-class or per-instance distribution behind the average, and a rank-stability comparison across defensible alternatives. When rankings change, the committee should resolve the objective conflict explicitly rather than select whichever column favors the incumbent model.
Calibration and external validity remain separate work
The framework operates mainly through confusion-matrix quantities. It can threshold probabilistic outputs and can represent soft labels through triangular norms, but it does not evaluate whether predicted probabilities are calibrated. Brier score, log loss, reliability analysis, and decision thresholds remain complementary requirements when confidence values affect escalation, pricing, abstention, or human review.
The empirical scope is also narrow: three synthetic studies and one multilabel dataset with four classifiers. The paper sketches how aggregation extends to costs, ordinal labels, uncertain ground truth, segmentation, detection, and cross-validation, but most of those settings are algebraic extensions rather than demonstrated production results. Its redundancy analysis is algebraic and ranking-based, not a study of statistical redundancy across many datasets. Full label-ranking evaluation and structured outputs also sit outside the present confusion-matrix formulation.
The paper supplies a disciplined way to expose weighting choices. It does not supply a universal metric. For operational evaluation, that is the more valuable outcome: the averaging selector becomes a declared decision rule, model ranking becomes a sensitivity question, and disagreement between scores becomes evidence about competing objectives that governance must settle.
Cognaptus: Automate the Present, Incubate the Future.