Bias on Demand: When Synthetic Data Exposes the Moral Logic of AI Fairness

The audit starts badly when everyone asks for “the fairness metric”

Audit.

That is where many AI fairness conversations become prematurely tidy. A model has produced uneven outcomes. Someone asks whether it is “fair.” Someone else proposes demographic parity, equal opportunity, calibration, predictive parity, or whatever metric most recently escaped from a conference paper into a compliance slide. The room nods gravely. A dashboard is born. Justice, apparently, has been converted into a ratio.

The dissertation On the Societal Impact of Machine Learning, by Joachim Baumann, is useful because it does not let that move pass unexamined.¹ It treats fairness not as a metric-selection problem, but as an operating problem with three linked stages: measure the unfairness, decompose the mechanism that produced it, and then choose an intervention that does not quietly make the system worse.

That sounds obvious. It is not. The common corporate version of AI fairness still assumes the hard part is picking a defensible metric and showing improvement against it. Baumann’s thesis keeps showing why that is too small. In insurance, the wrong metric can contradict the domain’s own moral logic. In synthetic data experiments, two datasets can look similarly biased while requiring different mitigation strategies. In ad delivery, fairness constraints can create “leveling down,” where the system achieves equality by making the better-served group worse off without meaningfully helping the worse-served group. Lovely. A fairness intervention that behaves like a bad spreadsheet compromise.

The central lesson is sharper than “fairness is complicated.” The lesson is that fairness is causal, normative, and operational at the same time. Remove any one of those three, and the analysis becomes theatre.

The thesis is really three instruments, not eight isolated papers

The dissertation brings together eight papers across fairness metrics, synthetic bias generation, feedback loops, online advertising, constrained decision rules, collective action in recommender systems, and rental-assistance allocation. A paper-by-paper summary would be accurate and almost completely unhelpful. It would produce a catalogue. The better structure is categorical, because the thesis itself is organised around a sequence of work that practitioners actually need.

The “bias on demand” paper is the conceptual hinge because it shows why the sequence matters. Synthetic data is not used here as fake data for cheap model training. It is used as a controlled laboratory for bias. The generator creates datasets with specific forms of bias, so researchers can observe how different mitigation techniques behave when the underlying data-generating mechanism is known.²

That is the luxury real systems deny us. In the real world, teams usually see symptoms: selection gaps, error-rate gaps, calibration drift, complaints, weird subgroup outcomes. They rarely see the exact mechanism that produced those symptoms. Synthetic data gives them a biased world in a jar. Not because jars are reality, but because reality rarely labels the bias for your convenience.

Measurement is where moral assumptions sneak into technical work

The first category of the thesis asks a deceptively simple question: what should fairness mean in this domain?

The insurance paper answers this by rejecting two metrics that are often treated as default candidates: independence, also known as demographic parity, and separation, often associated with equalised odds. The authors argue that neither is normatively appropriate for private insurance premiums under the assumptions they examine. Instead, they defend sufficiency, or well-calibration, as the relevant fairness criterion for detecting whether premiums systematically overestimate or underestimate risk for specific groups.³

This is not merely a technical preference. It follows from the structure of insurance. Insurance pricing is not a binary decision like “approve” or “deny.” It estimates expected risk and turns that estimate into a premium. If two groups have different average claim risks under the relevant insurance assumptions, demographic parity can become a demand to ignore risk differences that the pricing system is built to represent. Separation is also poorly matched because realised claims in a single year are not the same as individual risk. A policyholder who has no claim this year does not magically have zero risk. Insurers, regrettably, are not allowed to price metaphysics.

The paper’s practical example uses the freMTPL2freq dataset with 678,013 third-party liability policies. A group-blind Poisson GLM showed a systematic disadvantage for owners of cars younger than ten years. A group-aware version removed that systematic difference and slightly improved average unit Poisson deviance from 0.59 to 0.58. That example is not a general proof that adding sensitive attributes always improves fairness. The paper explicitly treats it as a demonstration of how sufficiency can detect systematic group disadvantage in premiums.

For business readers, the message is uncomfortable but important: “do not use protected attributes” is not the same as “do not discriminate.” A blind model can still encode group differences through other variables. Sometimes awareness is needed to test or reduce unfairness. Whether that is legally permissible or institutionally acceptable is a separate governance question. The model will not solve the lawyer’s problem. The lawyer, for once, remains employed.

The second measurement paper generalises the problem. It proposes a four-component framework for group fairness metrics: the utility experienced by decision subjects, the relevant groups to compare, the claim differentiator that justifies some inequalities, and the pattern of justice used to define a fair distribution.⁴

That framework matters because fairness metrics often hide moral assumptions in implementation details. Statistical parity, for example, tends to assume that positive decisions carry equal value for everyone. In many settings, that is false. A medical treatment may help one group and harm another. A loan may be beneficial for someone who can repay and harmful for someone pushed into default. A job recommendation may be valuable only if it is reachable, relevant, and not a dead-end funnel into low-quality work.

So the thesis pushes measurement away from metric shopping and toward explicit normative modelling. That is the first business lesson: fairness audits should begin with a moral specification, not a dashboard template.

Synthetic bias shows why the same symptom can need different medicine

The synthetic-data paper formalises several bias types, including historical bias, measurement bias, representation bias, and omitted-variable bias. It then generates controlled datasets where these biases can be varied and studied. This is the article’s title case because it makes the moral machinery visible.

The paper’s core distinction is between the world as it is observed and the world as one believes it should be interpreted. The authors discuss moral worldviews such as “what you see is what you get” and “we are all equal,” along with a more nuanced potential-space view. These are not decorative philosophy. They determine whether a group difference in the data is interpreted as legitimate variation, historical disadvantage, or measurement failure.

Consider the college admissions example. Suppose SAT scores differ across groups. If those scores are treated as faithful measures of skill, the disparity may be interpreted as historical bias affecting actual preparation. If the scores are treated as flawed proxies for skill, the same observed disparity becomes measurement bias. The numbers can look similar. The moral diagnosis changes.

The experiments then show that mitigation techniques behave differently depending on which bias is actually present. For historical bias on a relevant feature, “fairness through unawareness” does little because group information is already embedded in the feature. Removing the protected attribute does not remove its traces. For measurement bias on features, however, the model may need group information to correct the proxy. Blinding the model can reduce accuracy and generate unfairness across the considered metrics.

The biased-label case is even more dangerous. In the financial lending example, the observed repayment label can itself be a biased proxy for true creditworthiness. The model is trained on the proxy, and mitigation is also applied to the proxy. When evaluation is performed against the true target in the synthetic setup, many post-processing techniques fail under label measurement bias because they correct error rates relative to the biased observed label, not the true one.

This is a practical governance problem. Many businesses do not have “true” labels. They have historical approvals, historical repayments, historical performance scores, historical complaints, historical service usage, or historical human decisions. These are not neutral facts. They are institutional residues. Training on them without asking how they were produced is not data science. It is archaeology with a GPU.

The synthetic experiments in the paper serve as main mechanism evidence, not field evidence. Their purpose is to isolate bias types that are hard to separate in real datasets. The supplementary and variant analyses support robustness and scenario coverage. They do not prove that every deployment will behave exactly as simulated. They prove something narrower and more valuable: mitigation is mechanism-dependent.

For a company, the operational lesson is simple: before deciding how to mitigate bias, ask where the bias lives. Is it in the target label? The feature proxy? The sampling process? The decision rule? The feedback loop after deployment? These are different failure modes. A single fairness metric can describe the smoke; it cannot identify the wiring fault.

Feedback loops turn one-time unfairness into system behaviour

The feedback-loop paper extends the diagnosis from static datasets to deployed systems. This is where many fairness audits look charmingly obsolete. They test a model at launch, declare the error-rate gap tolerable, and forget that the model is about to change the world it measures.

The paper classifies five feedback loops in ML-based decision pipelines: sampling, individual, feature, ML model, and outcome feedback loops.⁵ Each loop describes a different way a decision can flow back into the system.

A sampling feedback loop changes who appears in the data. If a speech-recognition product works poorly for non-native speakers, they may stop using it, leaving future training data even more skewed toward native speakers. A feature feedback loop changes observable features. In lending, a loan decision can affect a credit score. An outcome feedback loop changes the outcome itself. A higher interest rate for a borrower deemed risky may increase the chance of default, thereby validating the original risk prediction in a particularly irritating way.

The important shift is from “the model is biased” to “the system has dynamics.” In deployed AI, a model does not merely observe reality. It participates in producing the next version of reality. That is not philosophical fog. It is product analytics, credit allocation, moderation policy, hiring funnels, ad delivery, and recommender systems operating over time.

The thesis uses this taxonomy as decomposition infrastructure. It does not claim that all feedback loops are bad. In control theory, feedback can stabilise systems. The problem is unmanaged feedback. If the system reinforces under-representation, discourages participation, distorts labels, or shifts behaviour in ways the model then learns from, the fairness problem becomes temporal.

For businesses, this means fairness monitoring cannot be a one-off pre-deployment audit. It needs cohort tracking, subgroup retention analysis, label-quality monitoring, and simulation of downstream behavioural effects. Otherwise the system may pass the launch audit and fail slowly, which is the preferred failure mode of organisations that enjoy plausible deniability.

Interventions can backfire when the cost is quietly assigned to users

The online advertising paper is one of the clearest warnings in the thesis. It studies fairness in ad delivery through simulations, modelling what happens when different fairness constraints are enforced in opportunity-distributing advertising systems.⁶

The findings are not the kind regulators can paste into a heroic press release. Statistical parity generally carries a higher utility cost for platforms than predictive parity or equality of opportunity. Enforcing one fairness criterion can worsen outcomes under another. When one group is more expensive to advertise to, fairness enforcement can lead to leveling down: the platform satisfies equality by reducing delivery to the better-served group rather than improving delivery to the worse-served group.

That distinction matters. A platform can make a metric look fair by reducing opportunity for one group without expanding opportunity for another. The dashboard improves. Society does not. Congratulations, the algorithm has discovered austerity.

The paper’s recommendation is not “do nothing.” It is that fairness policy must specify who bears the cost. The authors show that negative effects can be prevented when the platform enforces a fixed number of high-stakes ad impressions in both constrained and unconstrained scenarios. In that design, the platform carries the cost rather than passing it to users. In practice, platforms could still pass costs to advertisers unless regulation prevents it.

This has direct business relevance beyond advertising. Every fairness intervention redistributes cost somewhere: platform margin, advertiser spend, user experience, false positives, false negatives, manual review burden, or opportunity allocation. If governance does not specify the cost bearer, the system will usually choose the least politically visible one. That is rarely the disadvantaged group’s lucky day.

Fixing fairness requires knowing which lever is actually available

The intervention papers in the thesis illustrate three different control points.

The first is direct control over the decision rule. In the constrained-optimisation paper, Baumann, Hannák, and Heitz derive optimal decision rules for positive predictive value parity, false omission rate parity, and sufficiency.⁷ The counterintuitive result is that, under some conditions, the optimal rule for PPV or FOR parity can be an upper-bound threshold for one group: selecting individuals with the smallest utility rather than the most promising ones. For sufficiency, the decision rules become more complex and can introduce within-group unfairness for all but one group.

That result is not a curiosity. It is a warning that group fairness constraints can conflict with individual fairness. A rule can equalise a group-level metric while treating individuals within a group in ways that look perverse. This is exactly why fairness governance cannot stop at “we satisfied the constraint.” The question is what the constraint forced the system to do.

The second control point is user leverage over recommender systems. The collective-action paper studies fans who coordinate to promote an underrepresented artist by strategically placing a target song in playlists used for training a transformer-based music recommender.⁸ Using a public model released by Deezer and Spotify playlist data, the paper finds that small collectives controlling less than 0.01% of training data can produce up to 40 times more recommendations than average songs with comparable training frequency. The recommendations of other songs are largely preserved, and the strategy is framed as authentic playlist modification rather than fake-profile poisoning.

This is a different kind of intervention. The users do not control the model. They control part of the data. That matters in platform markets where affected communities cannot compel the platform to change its optimiser. Collective action becomes a data lever.

The third control point is institutional design in public services. The rental-assistance paper, developed with Allegheny County’s Department of Human Services, uses administrative data to prioritise proactive outreach to people facing eviction who are at high risk of homelessness.⁹ The model outperformed simpler prioritisation baselines by at least 20%, performed about ten times better than random selection, and could identify 28% of individuals overlooked by the current process who later used homelessness services. A shadow-mode deployment from September 2022 to August 2023 found that 22 of the top 100 predicted individuals used homelessness services within twelve months, consistent with the historical precision@100 around 0.20.

This is the strongest practical deployment story in the dissertation, but also the one with the clearest humility. The authors discuss eligibility constraints, label bias, data leakage, the difficulty of predicting first-time homelessness, and the need for field validation. The 2025 dissertation update reports that the system moved into active deployment, generating weekly lists of about 30 high-risk individuals for proactive outreach, with a randomized controlled trial launched in early 2025.

The business analogy is not “copy this model.” The useful lesson is workflow design. The team did not begin with “what interesting model can we train?” It began with a real institutional action: rental assistance outreach under resource constraints. The model’s evaluation metric, precision@100, matched the capacity constraint. That is what AI governance looks like when it is attached to an actual operating process instead of a committee memo.

The business pathway is governance architecture, not metric compliance

The thesis suggests a practical pathway for organisations deploying consequential AI systems.

First, define the decision and the affected utility. A positive classification is not automatically good. A loan, treatment, ad impression, premium, recommendation, or outreach intervention has different meanings depending on who receives it and under what conditions.

Second, choose the fairness criterion through domain reasoning. The metric should follow from the institutional purpose and the harms at stake. Insurance pricing may require calibration logic. Resource allocation may require opportunity-oriented recall. Recommender systems may require exposure and visibility measures. The point is not to be eclectic. The point is to stop pretending that one metric is portable across moral contexts like a USB-C cable.

Third, decompose the bias mechanism. Use causal diagrams, synthetic data, temporal validation, feedback-loop analysis, and subgroup monitoring to identify whether the observed disparity comes from features, labels, sampling, model training, decision rules, or post-deployment dynamics.

Fourth, select the intervention based on the available lever. A model owner can change the decision rule. A regulator can mandate audits and cost-bearing constraints. A platform community may use collective data leverage. A public agency can redesign outreach workflows. These are not interchangeable.

Fifth, monitor the system after deployment. Feedback loops, label drift, gaming, and changing population behaviour are not afterthoughts. They are the system.

The implied return on investment is not “fairer AI” in the abstract. It is fewer wrong interventions, fewer symbolic audits, fewer regulatory surprises, and better diagnosis when the system misbehaves. The cost is that organisations must make their value judgments explicit. Naturally, this is the part everyone will try to outsource to a metric.

The boundaries are real, and they matter

The dissertation’s evidence base is broad, but its practical reach has boundaries.

Most of the work focuses on group fairness in decision-making and recommender systems. It does not cover every notion of fairness, such as individual fairness or counterfactual fairness, in equal depth. Many approaches also require access to protected-attribute data, which may be legally restricted, poorly recorded, or politically sensitive.

Several results are theoretical or simulation-based. That is not a flaw; it is what allows mechanism isolation. But simulation evidence should not be mistaken for deployment evidence. The synthetic-bias generator shows how mitigation behaves under known mechanisms. Real data rarely announces its mechanism. The ad-delivery simulation clarifies trade-offs, but actual auction systems may have additional constraints and strategic responses.

The intervention work also depends on control rights. Post-processing assumes the decision-maker can adjust decision rules. Collective action assumes users can control some training data and that the recommender architecture is sequence-aware. The rental-assistance model is tailored to Allegheny County’s data, population, service infrastructure, and operational capacity. Its method may travel; its parameters should not.

Finally, fairness itself is not the only societal-impact issue. Privacy, consent, contestability, institutional accountability, and power asymmetries remain outside the metric frame. This is especially important as the thesis looks toward general-purpose and generative AI systems, where use cases are less bounded and affected groups may be harder to define.

Fairness is a pipeline, not a slogan

The lasting value of Baumann’s dissertation is that it makes AI fairness operational without making it simplistic. It does not offer the comfort of a universal metric. It offers a sequence: measure, decompose, fix.

That sequence matters because each stage prevents a specific organisational failure. Measurement prevents arbitrary metric selection. Decomposition prevents symptom treatment. Intervention design prevents fairness constraints from becoming expensive theatre or, worse, quiet harm.

The synthetic-data work gives the whole thesis its most memorable image: bias on demand. Build the biased world yourself, and you can finally see what the mitigation is really doing. The irony is useful. In real business systems, bias is rarely on demand. It arrives embedded in labels, incentives, proxies, user behaviour, feedback loops, and legacy processes. It arrives wearing normal clothes.

The job, then, is not to ask whether the model is fair in some abstract sense. The job is to ask what moral assumption the metric encodes, what mechanism produced the disparity, what intervention is actually available, who pays for it, and how the system will change after the intervention.

A dashboard can help with that. It cannot do the thinking for you. Shame. That would have been convenient.

Notes

Cognaptus: Automate the Present, Incubate the Future.