Unsupervised, Unaware, Unfair: When Your Embedding Knows Too Much

Segmentation is where many businesses go to feel mathematically innocent.

No target label. No credit decision. No hiring decision. No explicit age column. Just customers grouped by behavior, employees mapped by survey responses, users visualized in an embedding dashboard, or applicants compressed into a neat latent space before the “real” model begins.

This is the soothing story: if the sensitive column is removed, the representation cannot know it.

The paper SOMtime the World Ain’t Fair: Violating Fairness Using Self-Organizing Maps quietly ruins that comfort.¹ Its central result is not merely that age or income can be predicted from proxy variables. That point is already familiar in supervised fairness debates. The sharper point is geometric: an unsupervised representation can organize data along a withheld sensitive attribute before any downstream classifier, recommender, ranking model, or policy rule enters the room.

That is a different kind of failure. It is not “the model learned discrimination.” It is “the map rebuilt the sensitive axis, and everyone downstream inherited it.” Very efficient. Also inconvenient.

The misleading comfort of removing the sensitive column

The usual business version of fairness-through-unawareness is simple:

We removed age, gender, race, or income, so the model cannot use it.

For supervised prediction, this was never a strong argument. Proxy variables can reconstruct sensitive attributes: location, education, occupation, consumption patterns, device behavior, and response styles can all carry demographic structure. The more interesting problem appears one layer earlier.

Modern AI systems often use unsupervised representations before any explicit decision rule. These representations support:

• customer segmentation;
• behavioral clustering;
• employee analytics;
• recommender pre-processing;
• dashboard visualization;
• feature compression before supervised modeling.

In those settings, the organization of the data is already an operational artifact. A segment is not “just a segment” if it is mostly a disguised age band. A cluster is not neutral if its geometry quietly follows income. A dashboard is not harmless if managers interpret an embedding as behavior while it is partly sorting people by demographics.

The paper studies exactly this upstream layer. The sensitive attribute is withheld during training. It is used only afterward for auditing. The question is not whether a supervised attacker can predict age. The question is whether age or income becomes a dominant organizing principle of the unsupervised geometry itself.

That distinction matters. A probing classifier tells us whether sensitive information is extractable. SOMtime asks whether sensitive information has already become visible structure.

SOMtime is not a fairness cure; it is a leak detector

The paper introduces SOMtime as an auditing lens built on high-capacity Self-Organizing Maps, or SOMs. A SOM maps high-dimensional observations onto a structured lattice of prototype vectors. Nearby units on the lattice are trained to represent nearby regions of the input space, so the map preserves topology rather than merely compressing variance.

The authors then convert the SOM output into a three-dimensional embedding:

• two dimensions come from the lattice coordinates of the best-matching unit;
• the third dimension comes from quantization error, or distance from the observation to its prototype.

This is not presented as a mitigation method. SOMtime does not make the representation fair. It makes a particular kind of unfairness visible.

The mechanism is the article’s real subject. If we simply say “SOMtime gets higher correlations,” we miss the reason the result is interesting. The problem is not only leakage. It is leakage with shape.

Sensitive attributes in tabular business data often do not appear as one obvious proxy column. They are distributed weakly across many variables. Age may correlate slightly with moral survey responses, employment features, family structure, education, or purchasing patterns. Income may appear indirectly through occupation, working hours, capital gains, household characteristics, or other variables that are not labeled as income. Individually, these features may look safe enough. Collectively, they can form a gradient.

SOMs are good at turning those distributed local regularities into spatial organization. That is precisely why they are useful for visualization and clustering. It is also why they are dangerous if nobody audits the map.

Why PCA, UMAP, t-SNE, and autoencoders can miss the same signal

The paper compares SOMtime with common unsupervised methods: PCA, UMAP, t-SNE, and autoencoders. The important point is not that those methods are “bad.” They answer different geometric questions.

PCA looks for directions of high global variance. If the sensitive signal is low-variance but consistent, PCA can suppress it. In business terms, PCA may preserve the biggest behavioral difference while discarding a demographic gradient that is operationally sensitive but statistically quieter.

UMAP and t-SNE focus heavily on local neighborhood structure and visualization-friendly embeddings. They can separate clusters, but they are not designed to preserve global monotonic order. A sensitive attribute can be locally present yet globally scrambled, which makes it harder to see as a clean axis.

Autoencoders optimize reconstruction. They may preserve sensitive information in a diffuse latent form without concentrating it into a single interpretable dimension. A downstream supervised model might still extract it, but the embedding itself may not show a clean demographic path.

SOMtime’s advantage comes from a different inductive bias. The fixed lattice and competitive learning dynamics encourage local similarity to become spatial order. Weak proxy signals distributed across many input dimensions can become a coherent surface-level gradient.

That is the mechanism. The map does not need the age column. The remaining variables are enough breadcrumbs. SOMtime simply arranges the crumbs.

What the experiments actually test

The paper uses two real-world tabular datasets.

The first is the World Values Survey, using responses from Canada, Romania, Germany, China, and the United States. The sensitive attribute tested there is age. The second is Census-Income, where the authors examine age, income, and capital gains after removing the target sensitive attributes and also removing some strongly correlated columns to avoid making the task trivial.

The experimental design matters because it separates several claims that are easy to blur.

This is a useful paper because it does not stop at “there is correlation.” It asks whether the representation forms a usable sensitive topology. That is closer to how unsupervised embeddings create risk in practice.

A customer cluster is not audited by asking only, “Can a classifier recover age from this?” It should also be audited by asking, “Are customers arranged along an age gradient that marketing, pricing, or product teams will unknowingly act upon?”

The magnitude is large enough to be operationally boring — which is bad

The headline numbers are not subtle. In the paper’s reported Spearman correlations, SOMtime reaches 0.85 for age in the WVS Canada subset, 0.73 for WVS Germany, 0.83 for age in Census-Income, and 0.69 for income in Census-Income. The comparable baseline values are much lower in the highlighted cases: for Census age, the strongest baseline Spearman value reported is 0.22; for Census income, it is 0.25.

A small correlation might be an academic curiosity. These are not small enough to remain comfortably academic.

The sign is less important than the absolute magnitude: a negative value still means alignment, just in the opposite direction. The operational question is not whether older users appear on the left or the right of the map. It is whether the map has rebuilt age at all.

The authors also report trajectory recovery accuracy for the topology of sensitive groups. For WVS age, recovery accuracy is 1.00 in Canada, Romania, and Germany; 0.60 in China; and 0.80 in the United States. In Census-Income, they report 0.82 for age and 0.76 for income. Capital gains is much weaker at 0.22.

That last number is not a footnote to hide. It is a boundary. SOMtime is not a magic demographic extractor. It appears strongest when the withheld attribute has ordinal structure and enough distributed proxy signal in the remaining features. When the signal is weaker or less geometrically ordered, recovery can fail or become much less useful.

This is exactly why the paper is relevant for governance. It does not say “all embeddings are always unfair in the same way.” It says “some unsupervised embeddings can organize the world along sensitive axes even when you removed the obvious column.” That is already enough trouble for one afternoon.

The autoencoder ablation answers the obvious objection

The obvious objection is that SOMtime leaks more because the SOM has more capacity. Larger representation, more information retained, more leakage. Nothing mysterious.

The paper anticipates this by testing autoencoders at different sizes and bottleneck dimensions. The appendix reports small, medium, and large autoencoders, including large models with about 1.4 million parameters. Some large autoencoder settings achieve very low reconstruction error, yet their per-axis sensitive-attribute correlations remain far below SOMtime.

For example, the large 22-dimensional autoencoder reports Spearman correlations of 0.34 for WVS Canada age, 0.29 for WVS Germany age, 0.21 for Census age, and 0.17 for Census income. SOMtime reports 0.85, 0.73, 0.83, and 0.69 in the corresponding cases.

This is not a perfect proof that capacity never matters. Of course capacity matters. A tiny representation cannot preserve much. But the ablation weakens the lazy explanation that SOMtime wins simply because it is larger.

The better interpretation is architectural. Autoencoders can reconstruct data while spreading sensitive information across latent coordinates. SOMs, because of their lattice and competitive learning rule, concentrate distributed structure into geometry. Same ingredients, different cooking method. The smoke alarm goes off in one kitchen first.

The business risk begins before prediction

For a business reader, the practical lesson is not “stop using unsupervised learning.” That would be dramatic, expensive, and probably ignored by Tuesday.

The lesson is to stop treating unsupervised representation as a neutral preprocessing layer.

Consider four common uses.

The subtle risk is that unsupervised outputs often look less consequential than final model predictions. A cluster label feels descriptive. An embedding feels exploratory. A visualization feels like analysis, not action.

But organizations act on analysis. They assign sales teams to segments. They create differentiated campaigns. They route users into experiences. They prioritize accounts. They allocate attention.

If the representation has already rebuilt age or income, those business actions may inherit demographic structure without any explicit discriminatory rule.

This is why representation-level auditing should happen before downstream modeling. Waiting until the final classifier is audited may be too late. By then, the pipeline has already converted sensitive structure into “behavioral” features with better office manners.

What an audit should look for

The paper’s contribution is not a complete governance framework, but it points toward one. A practical audit should distinguish three levels of leakage.

First, test axis-level correlation. Are individual embedding dimensions correlated with known sensitive attributes? This is simple, interpretable, and cheap.

Second, test local group concentration. Do neighborhoods or clusters have demographic imbalance? This matters because cluster-based business actions often operate locally.

Third, test global ordering. Does the representation contain a monotonic path aligned with an ordinal sensitive attribute such as age or income? This is the SOMtime-specific insight. A representation can be dangerous not only because groups are separable, but because the entire map has become a demographic ruler.

These tests should be used differently depending on the deployment context.

For exploratory dashboards, the risk is interpretive. Analysts may see “behavioral structure” and assign a business story to what is partly demographic structure.

For preprocessing pipelines, the risk is inheritance. A downstream model trained on the embedding may use the sensitive geometry without needing the original sensitive column.

For segmentation systems, the risk is policy translation. A cluster becomes a pricing tier, product treatment, marketing message, or service priority. At that point, unsupervised structure is no longer harmless.

What the paper does not prove

The paper’s limits are important, and they are not decorative.

First, the strongest framing applies to ordinal sensitive attributes. Age and income have natural order. Spearman correlation and monotonic trajectory recovery make sense there. Categorical attributes such as race or gender would require different metrics. A race-sensitive topology might appear as separation, neighborhood concentration, or manifold branching rather than a clean monotonic path.

Second, the experiments are on tabular datasets. The results should not be casually pasted onto image, text, graph, or foundation-model embeddings without additional evidence. The mechanism may generalize in spirit — distributed proxy signals can become geometry — but the empirical claim here is narrower.

Third, SOMtime is an audit lens, not a mitigation. Revealing leakage does not remove it. The paper discusses possible future directions such as regularizing topology-preserving maps, penalizing sensitive gradients, or enforcing demographic entropy in neighborhoods, but those are not delivered as validated production tools.

Fourth, the evaluation uses known sensitive attributes for post-hoc auditing. That is normal and necessary for fairness work, but it means governance teams still need lawful, secure, and policy-compliant access to sensitive labels for audit purposes. The old trick of refusing to collect sensitive attributes can make discrimination harder to measure, not easier to avoid. Very elegant, in the way a blindfold is elegant.

The governance layer should move upstream

The most useful operational shift is simple:

Do not audit only decisions. Audit representations.

This changes the checklist. A company using embeddings in sensitive or semi-sensitive workflows should ask:

• Which columns were removed, and which proxy families remain?
• Are embedding axes correlated with age, income, or other regulated attributes?
• Do clusters show demographic imbalance?
• Does the embedding contain monotonic sensitive gradients?
• Are downstream teams using segments as if they were behavior-only groups?
• Does a mitigation step reduce leakage without destroying legitimate utility?

The last question matters because the goal is not to erase all structure. In real business data, age and income may correlate with legitimate needs, preferences, and constraints. Fairness governance is not solved by pretending those correlations do not exist. It is solved by deciding which uses are legitimate, which are prohibited, and which require controls.

That requires measuring the representation first.

SOMtime’s value is that it gives auditors a way to see a kind of structure that common projection methods may understate. If PCA and UMAP look clean while a topology-preserving map reveals a sensitive gradient, the correct conclusion is not “PCA made it fair.” It may simply have made the leak harder to see.

The embedding is already part of the decision system

The paper’s deeper lesson is that modern AI pipelines distribute decision-making across layers. We still talk as if the final model is the only place where fairness happens. That is increasingly obsolete.

The embedding shapes what the model can easily learn. The cluster shapes what the business team sees. The dashboard shapes what managers believe. The segment shapes how customers are treated.

So when an unsupervised representation rebuilds a withheld sensitive attribute, the organization has not avoided fairness risk. It has merely moved the risk into a layer that receives less scrutiny.

That is the uncomfortable contribution of SOMtime. It does not accuse the downstream model. It points at the geometry and says: the sensitive axis is already here.

The practical question is therefore no longer:

Did we remove the sensitive column?

It is:

Did the representation reconstruct it anyway?

A mature AI governance process should be able to answer that before the embedding becomes product infrastructure. Otherwise, the company is not practicing fairness-through-unawareness. It is practicing fairness-through-not-looking.

Cognaptus: Automate the Present, Incubate the Future.