Pick the Mistake Before You Pick the Metric

TL;DR for operators

A clustering score is not a neutral verdict. It is a policy for deciding which mistakes count.

Pasi Fränti’s review of external clustering measures separates that policy into three choices: how predicted clusters are matched to reference clusters, how similarity is scored, and how results are normalized.¹ Those choices determine whether the metric rewards getting many individual records right, getting each cluster right regardless of size, or locating the correct cluster structure.

The operational recommendations are straightforward:

The paper directly supports these distinctions through a taxonomy of metric design, constructed matching examples, and comparisons drawn from earlier clustering benchmarks. Cognaptus’ business inference is that metric selection should become part of the evaluation specification, not an afterthought added once the model has already produced an attractive score.

The main boundary is equally simple: external metrics require a credible reference clustering. If the “ground truth” is politically negotiated, historically stale, or merely convenient, a more elegant metric will calculate the wrong answer with greater discipline.

Segment 7 Does Not Know It Is Segment 7

Suppose a customer-segmentation model produces five groups. The reference data also contains five groups. It is tempting to compare label 1 with label 1, label 2 with label 2, and continue until the spreadsheet becomes reassuring.

That would be wrong.

Cluster labels are arbitrary identifiers. A model’s “cluster 1” has no inherent relationship to the reference’s “cluster 1.” One solution might label the premium-customer group as 2 while another labels it as 5. Before accuracy can be calculated, the clusters must first be matched.

The difficulty becomes more interesting when the numbers of clusters differ. A predicted solution may split one reference cluster into three, merge two reference clusters into one, or invent a small orphan group. At that point, evaluation is no longer ordinary classification with eccentric labels. It is an assignment problem.

Fränti’s paper is useful because it does not treat the resulting collection of indexes as a list of formulas to memorize. It decomposes set-matching metrics into three design decisions:¹

1. Matching: Which predicted and reference clusters correspond?
2. Similarity: How much credit should a matched pair receive?
3. Normalization: Should influence be allocated by points, by clusters, or not normalized at all?

That decomposition is the paper’s most important contribution. It turns metric selection from brand recognition—“everyone reports NMI”—into an explicit choice about what an error means.

A score of 0.94 may look precise. Precision of presentation is not the same thing as clarity of interpretation.

Mapping and Pairing Reward Different Structures

The first decision is whether clusters are mapped or paired.

Under mapping, each cluster in one solution selects its best match in the other. The same target cluster can therefore be selected more than once. One-directional measures such as purity can consequently over-credit solutions when the number of clusters differs. If a model splits one real group into several predicted groups, each fragment may point back to the same reference cluster and receive substantial credit.

Pairing is stricter. Each predicted cluster can be paired with at most one reference cluster. Once a reference cluster has been claimed, another predicted cluster cannot quietly borrow it.

The paper’s Figure 8 uses a constructed example to make the distinction visible. Mapping counts 15 of 20 points as matched, producing 75%. Pairing counts 5 of 10 in the illustrated pairing setup, producing 50%.² The purpose of the example is explanatory rather than empirical: it isolates the consequence of permitting many-to-one matches.

That difference matters because splitting and merging are not cosmetic errors.

Imagine a document-clustering system that divides one coherent regulatory topic into four near-duplicate groups. A mapping-based score may conclude that each group resembles the correct topic. Operationally, however, reviewers must now search four queues, reconcile duplicate summaries, and wonder whether the fragmentation represents a real distinction. The score has recognized semantic overlap while ignoring workflow damage.

Pairing-based measures such as ACC and PSI prevent multiple predicted clusters from claiming the same reference cluster. They solve an optimal assignment problem, commonly with the Hungarian algorithm. The paper notes that the revised ACC formulation can handle unequal cluster counts by adding dummy clusters before pairing. The nominal $O(k^3)$ complexity is usually manageable because the number of clusters $k$ is typically far smaller than the number of records $N$.

The practical choice is not “mapping bad, pairing good.” Bidirectional mapping can be simpler and suitable when flexible correspondence is intended. The point is that the evaluator must know whether duplicated coverage should count as success.

If the metric allows three predicted clusters to receive credit for rediscovering one real cluster, the metric has taken a position on over-segmentation. It would be polite to notice.

Raw Overlap Makes Large Clusters Louder

Once clusters have been matched, the evaluator must decide how to score their similarity.

ACC and several related measures use the number of shared points:

This is intuitive. If a predicted cluster and its reference counterpart share many records, they are similar. Summing correct matches and dividing by $N$ produces a point-weighted accuracy score.

The consequence is equally intuitive but often ignored: large clusters contribute more because they contain more points.

Suppose 95% of customers belong to one broad low-value segment, while the remaining 5% occupy several small but commercially important groups. A model can score well under a point-weighted measure by handling the large group correctly while confusing the smaller groups. The metric is not malfunctioning. It is faithfully implementing the rule that each customer, rather than each segment, receives equal weight.

That rule may be exactly right. If each incorrectly routed record creates roughly the same service cost, point weighting aligns with operational loss.

It may also be wrong. If a small cluster represents fraud, churn risk, a regulated population, or a high-margin account tier, its size is not a reliable measure of its importance.

PSI addresses the size issue by applying the Braun–Banquet similarity to paired clusters:

Each paired cluster receives a score in $[0,1]$, and the result is normalized by the number of clusters rather than by the number of points. This gives clusters more equal influence even when their sizes differ.

The contrast is therefore not primarily mathematical. It is an allocation decision:

• ACC: every point matters equally.
• PSI: every cluster matters more equally.
• CI: the primary object is whether the cluster itself was structurally located.

There is no universally fair weighting scheme hiding behind the notation. There is only a weighting scheme that matches—or fails to match—the operating objective.

CI Counts Broken Clusters; ACC Counts Misassigned Points

The paper’s strongest recommendation is the centroid index because its output is directly interpretable.¹

CI matches centroids in both directions and counts unmatched centroids, sometimes described as orphan centroids. Its score runs from 0 to $k$:

• $\operatorname{CI}=0$ means the cluster structure has been correctly located at the cluster level.
• $\operatorname{CI}=7$ means seven cluster locations are wrong.
• A relative form can divide by $k$ to express the proportion of cluster-level errors.

This is unusually explainable by clustering-metric standards. An NMI of 0.91 invites another meeting about whether 0.91 is acceptable. A CI of 7 invites someone to inspect seven failures.

That diagnostic convenience does not make CI a finer metric. It makes it a coarser metric with a clearer unit.

Figure 10 establishes the boundary. In the paper’s Birch2 example, $\operatorname{CI}=7$ corresponds to seven misplaced centroids. In the S3 example, $\operatorname{CI}=0$ because all centroids are roughly in their correct locations, even though slight centroid deviations create visible point-level disagreement around cluster boundaries.³

This figure is not a robustness test or a second empirical thesis. Its purpose is to demonstrate exactly what CI refuses to measure.

That refusal can be appropriate. During algorithm development, a team may first need to know whether the model discovered the correct number and broad placement of customer groups. Cluster-level diagnosis separates structural failure from local assignment noise.

Later, when the system assigns individual customers to treatments, boundary errors become operationally significant. ACC or another point-level measure is then more informative.

The two metrics answer different questions:

Using both is often more informative than forcing one score to perform both jobs. A model with $\operatorname{CI}=0$ but mediocre ACC has found the broad structure while drawing poor boundaries. A model with low point error but nonzero CI may be hiding structural mistakes inside a dominant mass of correctly assigned records.

That two-level reading is a Cognaptus inference from the paper’s distinction, not a benchmarking protocol directly tested by the paper. It is nevertheless the kind of inference one can make without requiring interpretive acrobatics.

NMI Can Look Excellent While Eighteen Clusters Are Wrong

Normalized mutual information is popular because it produces a bounded score, sounds theoretically respectable, and appears in enough clustering papers to feel compulsory.

The paper’s Figure 9 supplies a useful antidote to that comfort. It compares NMI and CI values for several algorithms on seven selected benchmark datasets from prior work. Clustering results judged visually correct receive NMI values between 0.93 and 1.00. Unfortunately, some incorrect results also receive high values.⁴

The sharpest example is Birch2. One result has 18 incorrectly located clusters out of 100, giving $\operatorname{CI}=18$, while NMI reports 0.96.

A score of 0.96 ordinarily reads as “nearly solved.” Here it coexists with structural errors in 18% of the clusters.

The paper attributes the result to a documented NMI bias that becomes especially relevant when the number of clusters is large. It also points to incorrect S-series results scoring 0.93 and 0.95. The implication is not that NMI contains no information. It is that its magnitude is difficult to interpret and may not be comparable across datasets with different clustering structures.

Figure 9 is the closest the paper comes to headline empirical evidence, but its purpose should be kept straight. The numbers are selected results from earlier benchmark work, used here to illustrate interpretability and bias. They are not a new, comprehensive tournament covering every metric, data geometry, or business setting.

ARI has a different problem. It adjusts the classical Rand index for the agreement expected under random clustering, making it considerably more useful than unadjusted RI. Yet when cluster sizes are unbalanced, ARI can primarily reflect agreement among the largest clusters.

Classical Rand index is worse. It examines whether pairs of points are consistently placed together or apart. Because most pairs in a multi-cluster dataset belong to different clusters, the index can be overwhelmed by easy “different-cluster” agreements. The result is often high even for clustering solutions that are not especially good. The paper therefore does not recommend it.

The broader correction is this:

Familiar metrics are not common currencies. They are financial instruments with undocumented exposure.

A high NMI, ARI, or RI score does not automatically mean that the clustering is nearly correct. It means that the solution performed well under that metric’s accounting rules.

The Paper Is a Metric Design Guide, Not a New Leaderboard

The paper combines several kinds of material. Treating them as interchangeable evidence would make the review appear either stronger or weaker than it is.

There are no ablations in the usual machine-learning sense because the paper does not introduce a multi-component predictive model. There is also no appendix battery establishing robustness across dozens of alternative conditions. The contribution is synthetic and methodological: the author organizes existing metrics, explains their mechanics, demonstrates their failure modes, and gives recommendations.

That is valuable, but it is a different kind of value from a large new benchmark. The paper improves the evaluator’s vocabulary rather than claiming to have settled every possible metric dispute.

A Metric Contract for Business Clustering

The paper directly shows that metric choice changes what receives credit. The business consequence is that teams should define the evaluation policy before comparing models.

A minimal metric contract should answer four questions.

What is the unit of harm?

Is a mistake costly because one record is assigned incorrectly, because one business segment disappears, or because a small but critical group is ignored?

• Per-record cost points toward ACC.
• Per-cluster structural cost points toward CI.
• Equal concern across differently sized clusters points toward PSI.

Are splits and merges operationally different?

A system that splits one real segment into three may preserve semantic content while creating duplicated workflows. A system that merges three segments into one may simplify operations while destroying differentiation.

Because mapping and pairing handle these structures differently, teams should report split and merge patterns rather than relying exclusively on a single aggregate score.

Is the reference clustering actually authoritative?

External evaluation assumes that the ground truth is meaningful. In synthetic benchmarks, this can be defensible because the data-generating clusters are known. In business datasets, reference labels may come from analyst judgment, legacy taxonomies, past campaigns, or downstream outcomes.

A metric cannot determine whether those labels represent reality. It can only measure conformity to them.

Must results be compared across datasets?

The paper explicitly questions the comparability of NMI magnitudes across datasets. That matters when a central analytics team reports one clustering score across regions, products, languages, or time periods.

Cross-dataset dashboards should prefer metrics with operational units or accompany normalized scores with structural context: cluster counts, size distributions, CI values, and error concentration. Otherwise, a regional score of 0.95 and a product score of 0.92 may be treated as comparable when the underlying evaluation problems differ substantially.

A practical evaluation stack might therefore contain:

1. CI or generalized CI for structural diagnosis.
2. ACC for point-level performance.
3. PSI when minority clusters require equalized visibility.
4. A split-and-merge inspection for operational interpretation.
5. Downstream application metrics to determine whether any clustering difference matters commercially.

The fifth item extends beyond the paper’s external-index recommendations. Fränti discusses application-level evaluation and notes that clustering may be only one component of a larger system. Cognaptus’ inference is that external validity should diagnose the clustering, while downstream metrics decide whether the diagnosis affects the business.

Confusing those roles produces two familiar mistakes: declaring a model useful because its clusters resemble a benchmark, or rejecting a model because its clustering score changed even though the downstream workflow did not.

Where the Recommendations Stop

The guidance is strongest when comparing hard partitions against credible ground truth.

Four boundaries materially affect practical use.

First, CI is intentionally insensitive to point-level differences. It can establish that clusters are in roughly the right places while overlooking assignment errors near their boundaries. It should not be used alone when record-level treatment depends on those assignments.

Second, the original CI is naturally suited to centroid-based clustering. The paper discusses generalized variants that replace centroid matching with overlap-based matching, but practitioners should not assume that every non-centroid clustering inherits the same interpretation without checking the implementation.

Third, ACC and PSI express different weighting policies. ACC does not accidentally favor large clusters; it gives each point equal influence. PSI does not discover business importance; it gives clusters more equal influence. Neither weighting is universally correct.

Fourth, the paper is a methodological review with selective illustrations. Its recommendations are reasoned from known metric properties, prior analyses, and benchmark examples. They are not the outcome of a new exhaustive experiment spanning text embeddings, graph communities, customer behavior, image clustering, and every fashionable latent space currently seeking budget approval.

The scope also does not establish how these recommendations transfer to fuzzy membership, hierarchical clustering, overlapping communities, or settings with several equally defensible reference partitions. Those are not minor implementation variations. They change what “ground truth” and “correct assignment” mean.

Stop Asking Which Score Is Best

The right clustering metric is not the one with the most citations, the neatest range, or the highest value on the current model.

It is the one whose errors correspond to the decision being made.

Use CI when the central question is whether the correct cluster structure was found and explainability matters. Use ACC when each record should count equally and point-level assignment is the object of evaluation. Use PSI when every cluster should retain influence despite imbalanced sizes. Use ARI and NMI with an understanding of their biases, not as ceremonial proof that the model has behaved. Leave classical Rand index to history unless there is a very specific reason to revive it.

Most importantly, report enough context that another person can tell what the score means.

A clustering metric is not a verdict delivered by mathematics. It is a compact statement about which mistakes the evaluator has chosen to forgive.

Choose the forgiveness policy first.

Cognaptus: Automate the Present, Incubate the Future.