Trust Issues at 35,000 Feet: Assuring AI Digital Twins Before They Fly

Airspace is a bad place to discover that your simulation was “mostly right.”

That sentence is obvious enough to sound useless, but it points to the real issue. For an AI-enabled digital twin of air traffic control, being “accurate” is not one property. It is a stack of claims. The data must be representative. The software representation must preserve the right details. The trajectory predictor must handle uncertainty rather than pretending aircraft behave like obedient geometry. The AI agents using the twin must receive, act on, and explain information without corrupting the control problem on the way.

A recent Project Bluebird paper, A framework for assuring the accuracy and fidelity of an AI-enabled Digital Twin of en route UK airspace, is valuable because it refuses to compress all of that into the lazy phrase “validated model.”¹ It uses the Trustworthy and Ethical Assurance, or TEA, methodology to build an assurance case for a probabilistic digital twin of en route UK airspace and NATS Basic Training environments. The digital twin is designed primarily as a training and testing environment for AI air-traffic-control agents, not as a certified operational controller.

That distinction matters. The paper is not saying: here is a digital twin, therefore trust it. It is saying: here is the structured argument we would need before trust becomes a defensible position. Less glamorous. More useful. A rare combination.

The misconception: model accuracy is only one slice of digital-twin trust

A common business reading of digital twins goes something like this: build a virtual copy of the real system, compare model outputs against historical reality, show acceptable error, and then use the twin for training, optimisation, or decision support.

That logic is not wrong. It is just underpowered.

Project Bluebird’s case shows why. Its digital twin is not a spreadsheet forecast. It ingests radar, clearances, flight plans, sector geometry, weather, and training-scenario data. It replays historical trajectories. It predicts future trajectories. It can generate synthetic air-traffic scenarios using large language models. It exposes APIs and gymnasium environments for reinforcement learning, rules-based agents, and optimisation agents. It also includes a human-machine interface for expert review.

At that point, asking whether “the model” is accurate becomes a suspiciously small question. Which model? Which data? Which representation? Which use case? Which aircraft type? Which manoeuvre? Which time horizon? Which agent interface? Which source of uncertainty? The paper’s top-level goal claim is therefore carefully scoped:

the Digital Twin of en route UK airspace and Basic Training has sufficient fidelity and accuracy for its intended uses.

The important words are not “accuracy” and “fidelity.” They are “sufficient” and “intended uses.”

“Sufficient” forces thresholds, evidence, assumptions, and justifications. “Intended uses” prevents the usual software-demo sin: proving a system works in one context, then quietly selling that proof into another.

The accepted structure for reading this paper is category-based because the paper itself is category-based. Its main move is to break assurance into four strategies:

That table is the article. The rest is unpacking why each row changes the evidence required.

TEA turns trust into a structured argument, not a brand mood

The paper uses TEA to construct an assurance case: a nested argument connecting a top-level claim to strategies, property claims, evidence, assumptions, and justifications. This is not merely documentation. It is a discipline for preventing teams from hiding uncertainty in prose.

In the TEA structure, a claim is not accepted because a team feels confident. It is accepted because evidence has been attached, the evidence is relevant to the claim, and the assumptions behind that evidence are visible enough for reviewers to challenge.

That visibility is especially important here because Project Bluebird sits in an awkward but familiar zone for frontier AI systems. Traditional verification and validation practices still apply to many software components. But some components do not fit cleanly into older assurance habits: probabilistic trajectory generation, live-data shadowing, LLM-generated scenarios, and AI agents interacting with a dynamic simulation environment.

The paper therefore does something more practical than proposing a new grand theory. It maps existing and emerging guidance onto a working digital twin. The authors draw from digital-twin principles, data-quality dimensions, AI/ML assurance guidance in air traffic management, and established air-traffic-control practices. Then they translate that into property claims.

This is where the paper becomes useful outside aviation. Many firms are now building AI copilots, operational simulators, agent sandboxes, robotics testbeds, and “digital twins” of workflows. Most of them do not need aviation-grade assurance. But they do need the habit aviation is famous for: never trust the average case when the edge case is where people get hurt, money disappears, or regulators become suddenly literate.

Layer 1: the data pipeline is where assurance becomes boring on purpose

The first assurance strategy concerns the data pipeline. This is the least fashionable part of the paper and probably the most transferable.

Project Bluebird uses historical operational data from air-traffic systems, radar track data, flight plans, clearances, coordination records, weather data, and Basic Training exercise data. The paper notes that historical sources go back to 2016 and include more than 20 million flights. Basic Training data covers exercises from 2024 to 2025. Live data streaming is also being developed for real-time shadowing.

A lesser paper would stop there and say “large-scale real-world data.” The authors do not. They ask whether the curated data has sufficient fidelity, relevance, quality, and integrity for the intended uses.

That breaks into several practical questions:

• Is the data inside the operational domain?
• Is it complete enough for the simulation task?
• Is the resolution appropriate?
• Is it timely enough, especially for live shadowing?
• Has it been corrupted, duplicated, mislabeled, or cleaned incorrectly?
• Are outliers genuine aircraft behaviours, data errors, or wrangling failures?

One example is nicely concrete. A “descend when ready” clearance may be mislabeled as “descend now” because of default system behaviour. In simulation, that difference is not clerical. It changes when the aircraft descends, which changes the future traffic situation, which changes what an AI controller learns. Congratulations: a labeling shortcut has become a training signal.

Another example concerns callsigns. Two tracks may mistakenly share a callsign, creating apparently impossible speeds. A simplistic cleaning rule could remove valid data. The better approach is not “delete weird things.” It is to distinguish implausible physics from messy but explainable operational records.

For business readers, the lesson is blunt: data provenance is not a compliance appendix. It is part of the model.

In ordinary AI projects, teams often treat data engineering as a prerequisite and model evaluation as the real validation. This paper pushes the opposite habit. The pipeline itself needs property claims. The data’s source systems may have existing technical assurance, but that does not automatically prove the data is complete, usable, or fit for the digital twin’s purpose. Machine-readable data is not the same as AI-ready data. Regulators, and annoyed customers, eventually notice.

Layer 2: virtual representation is where realism gets quietly lost

The second strategy concerns the virtual environment: how curated data becomes a software representation of airspace, aircraft, events, weather, trajectories, and clearances.

This layer is easy to underestimate because it sounds like implementation. It is not. It is where the system decides which real-world details survive the translation into code.

The paper’s digital twin represents airspace and aircraft through object-oriented software classes. Replay mode uses historical radar and clearance data to recreate operational scenarios. That replay is not just a feature; it is an assurance instrument. By comparing replayed trajectories against curated input data, the team can isolate errors caused by the virtual representation before blaming the trajectory predictor.

That separation is editorially important. In many AI evaluations, error attribution is mushy. A poor result may come from bad data, bad preprocessing, poor model architecture, a brittle interface, or a flawed benchmark. The Project Bluebird framework forces error localization. If the replay is already wrong, the predictor is not the first suspect. If the replay is correct but the future prediction diverges, the predictor becomes more relevant.

The paper lists several metrics and error types for replay and virtual representation:

The paper is admirably specific about limitations. Holding patterns and conditional clearances are not yet supported. Some speed instructions such as “speed greater than” or “speed less than” are interpreted as “speed equals.” That is not a tiny UI detail. It is a declared loss of fidelity.

The practical point is not that every digital twin must capture every detail. That would be fantasy, and expensive fantasy is still fantasy. The point is that every omitted detail should have a known consequence. Approximation is acceptable when it is bounded, documented, and aligned with the use case. Approximation becomes dangerous when it is smuggled into the system under the name of engineering pragmatism.

Synthetic scenarios: the LLM is useful only after it is boxed in

The paper’s first deep dive concerns LLM-generated synthetic air-traffic scenarios. This is where the article could easily become fashionable and therefore less useful. Thankfully, the authors keep the LLM on a leash.

The digital twin can generate synthetic datasets to augment rare situations or test AI agents against unexpected scenarios. The process converts an airspace sector into a simplified grid representation and prompts an LLM to create aircraft configurations satisfying user-specified parameters: number of routes, duration, number of aircraft, and number of interactions.

The assurance question is not “can GPT-5 write a plausible scenario?” The question is narrower and more testable: can a specific LLM reliably generate the correct scenario for a given prompt, under benchmarked constraints?

The paper’s deep dive describes benchmarks where parameters are varied and averaged over 10 synthetic sectors. For traffic volume, aircraft count ranges from 2 to 30 while duration, routes, and intersections are held fixed. Similar benchmark logic applies to other prompt-controlled parameters. The authors also check whether benchmarked scenarios are solvable, so the LLM is not blamed for failing an impossible request.

This evidence is mainly a property-claim test, not a general endorsement of LLM reasoning. The reported result is that with a feedback loop, GPT-5 can meet the threshold across the benchmarked metrics, and it is close even without feedback. For some models, automated self-correction can achieve perfect scores across all non-interacting benchmarks. The authors therefore assume a 100% threshold for those benchmarks because faithful prompt interpretation is a necessary precondition for fit-for-purpose scenario generation without hallucination.

That threshold is severe, and correctly so. In a training environment, a slightly wrong synthetic scenario is not merely a content error. It can teach an agent the wrong distribution of operational difficulty.

The paper also distinguishes benchmark success from robustness. The benchmarked prompts are carefully curated. As prompts become more domain-specific — for example, “two aircraft in trail and a third crossing tracks” — automatic verification becomes harder. The authors therefore propose sensitivity analysis for prompt variations and human-in-the-loop evaluation by air traffic controllers using visual assessment and scoring.

This is a clean example of how to use LLMs in regulated or high-consequence simulation work:

1. constrain the output space;
2. benchmark directly against prompt-controllable parameters;
3. use feedback loops for correction;
4. test prompt sensitivity;
5. bring domain experts in when automatic checking becomes brittle;
6. do not pretend the foundation model’s training history has magically become your assurance evidence.

The business implication is simple. LLMs can be useful generators of test cases, but only if the generation system is itself tested as a component. “Synthetic data” is not automatically a cheap substitute for rare real-world data. It is an evidence-producing machine, and machines that produce evidence require their own evidence. Annoying, yes. Also how grown-up systems work.

Layer 3: trajectory prediction must prove uncertainty, not just average accuracy

The third strategy concerns trajectory prediction. This is the most technically dense part of the paper, and it deserves attention because it shows why deterministic accuracy is too small a target.

The digital twin uses several trajectory predictors. These include BADA, an industry-standard deterministic aircraft-performance model; ACE, the in-house predictor used in NATS’s ATCO training college; a probabilistic physics-informed machine-learning predictor; and a mean-mode version of the probabilistic predictor for deterministic decision support.

The key issue is epistemic uncertainty. In real air traffic management, controllers often do not know aircraft mass, pilot response time, or exact performance settings. A deterministic predictor can give one future path. But the real world gives a distribution of possible paths. A useful digital twin for training AI agents must capture that uncertainty, not airbrush it away.

The probabilistic predictor uses BADA as a physics-informed base and applies learned correction terms to quantities such as drag, thrust, and calibrated airspeed. The model is trained separately by aircraft type. The paper discusses Boeing 737-800 descending aircraft as a deep dive.

This deep dive is main evidence for uncertainty fidelity, not a full operational safety proof. The authors compare generated trajectory distributions against held-out real-world data using measures such as Kolmogorov-Smirnov distance and Wasserstein distance. They also examine relevant quantities such as calibrated airspeed, rate of descent, and time to bottom of descent.

The reported B738 descent comparison is concise:

The paper is careful about interpretation. There are no fixed universal thresholds for these distances. Unlike mean-mode prediction, where BADA can serve as a benchmark, uncertainty fidelity depends on the data distribution, sample size, aircraft type, and modality. The authors use visual similarity and model improvement as starting points.

That caution is not weakness. It is exactly the point of an assurance case. A metric without a defensible threshold is a measurement, not yet an argument.

For deterministic mean-mode prediction, the proposed evidence includes mean absolute error in along-track, cross-track, and vertical dimensions, compared against BADA. The paper also distinguishes look-ahead horizons: one radar blip, approximately six seconds, and the interval from one clearance to the next. This is a useful design choice because prediction error is not a scalar floating in space. It depends on the time horizon over which the system is expected to be useful.

For probabilistic inference, the paper proposes tools such as Continuous Ranked Probability Score and calibration curves. A calibration curve asks whether stated probability intervals contain reality at the expected rate. If the 95% interval contains the real trajectory only 70% of the time, the system is not merely “uncertain.” It is badly calibrated. That distinction matters for any AI assistant that presents confidence bounds, risk estimates, or what-if projections.

The broader business lesson is that uncertainty is part of the product. In operational twins, financial forecasting tools, supply-chain simulators, and clinical decision support, users do not only need the most likely path. They need to know how wrong the system can be, where it tends to be wrong, and whether its uncertainty estimates deserve trust.

Layer 4: AI-agent interoperability is part of the test, not plumbing around the test

The fourth strategy concerns AI agents as enabled by the digital twin. This is where many agent demos become quietly unserious.

The Project Bluebird digital twin is designed to be agent-agnostic. Agents connect through a REST API, exchange structured JSON messages, and receive information about aircraft positions based on replay or trajectory projections. The environment also supports gymnasium-style training and evaluation. Agents include reinforcement-learning, rules-based, and optimisation approaches.

That architecture creates a trap. If the agent performs badly, is the agent bad? Or did the API drop a message? Did a timestamp drift? Did latency exceed the control timescale? Did the schema allow an invalid clearance? Did the test harness distort the situation?

The paper’s Property Claim 9 addresses the two-way connection between digital twin and agents. Information relevant to decision-making must be transmitted completely and correctly. Messages must carry the right simulation timestamp. Partial failures such as dropped or duplicated packets must be guarded against. Latency should ideally fit within a radar update cycle of about six seconds. Validation therefore includes message integrity, throughput, round-trip latency under representative load, schema checks, authentication, authorization, secure coding, static analysis, contract testing, and performance testing.

That sounds like mundane systems engineering because it is. But for AI agents, mundane systems engineering is often the difference between an experiment and theatre.

The paper then shifts from connection fidelity to assessment. Agents can be evaluated on safety and efficiency metrics, including loss of separation, technical safety, coordination compliance, airspace excursions, fuel burn, and traffic complexity. Maintaining separation is non-negotiable. Other thresholds may be determined by comparison with experienced air traffic controllers, hypothesis testing, expert consultation, or learning from human-assigned Basic Training scores.

This is a useful correction to metric worship. A single numerical score cannot capture air traffic control proficiency because controllers balance safety, efficiency, orderliness, and context. The digital twin therefore includes an HMI that allows human experts to inspect agent behaviour in a familiar operational style.

Explanation is treated similarly. The goal is not to dump an agent’s internal decision tree onto an assessor. In real time, that would be less transparency than punishment. The paper discusses visualising plans, relevant aircraft, intended trajectories, clearances, and potentially counterfactual explanations. The right abstraction depends on the assessor’s expertise and time constraints.

For business use, this is one of the paper’s more transferable insights. An AI agent cannot be assured only by offline benchmark scores. It must be assured through the interface by which it sees the world, acts on the world, logs its choices, and explains those choices to people who know the domain better than the model does.

The two deep dives are evidence examples, not victory laps

The paper provides two detailed evidence deep dives: GPT-5 synthetic scenario generation and probabilistic B738 trajectory uncertainty. They serve different assurance purposes.

This distinction protects the paper from overclaiming. The deep dives are not decorations. They show how evidence should be attached to claims and how assumptions should be declared.

That is also why the paper’s style matters. It repeatedly makes assumptions explicit: CAS is treated as relevant because it is directly learned and proxies distance travelled; KS and Wasserstein distances are treated as complementary because they capture different aspects of distributional mismatch; prompt benchmarks are tied to controllable prompt parameters; a 20 nautical mile grid is a trade-off informed by controller experience and benchmark feasibility.

In weaker AI papers, assumptions hide in method choices. Here, they are first-class objects. This is not just scholarly neatness. It is operational hygiene.

What businesses should take from the framework

The immediate domain is aviation. The broader use case is any organisation trying to use AI inside a simulated representation of a complex system.

That includes warehouse robotics, logistics routing, energy grids, manufacturing lines, call-centre automation, claims processing, trading simulations, cyber ranges, and enterprise workflow agents. The details differ. The assurance pattern travels.

A practical business adaptation would look like this:

This is not cheap bureaucracy. It is cheaper diagnosis.

Without the layered structure, a failed agent test becomes a foggy post-mortem. With it, teams can ask whether the failure came from missing data, a replay approximation, a trajectory predictor, an LLM-generated training case, a latency issue, or a weak evaluation metric. That is ROI in the least glamorous sense: less time arguing about what broke.

The framework also makes procurement and stakeholder conversations more concrete. Instead of asking vendors whether their AI simulator is “validated,” a buyer can ask:

• What is the intended operational domain?
• Which components are approximated, and how is approximation error measured?
• How do you test synthetic scenario generation?
• How do you distinguish data-pipeline error from prediction error?
• Are uncertainty estimates calibrated?
• What happens when live data goes stale?
• How do agents receive state and issue actions?
• What evidence is automated, and what still requires expert assessment?

These questions are impolite in the healthiest possible way.

Boundaries: this is not certification, and the paper says so

The paper’s own limitations are important because they define how the work should be used.

First, the current assurance case focuses on accuracy and fidelity for training and testing AI agents. It is not a full safety case. Operational deployment would require more stringent thresholds and additional evidence, including failure-mode analysis, safety envelope testing, and risk-oriented measures. In aviation, the difference between “good enough for a training environment” and “good enough for operational use” is not a footnote. It is the plot.

Second, the assurance case is still a living document. The paper provides an overview and selected deep dives, not a complete published case. That is reasonable for research, but business readers should not confuse the framework with completed assurance.

Third, live data assimilation remains a harder problem. Live shadowing requires online validation, freshness monitoring, latency control, and assurance of extrapolation methods. The paper notes that real-time learning remains difficult under current certification processes. In other words, the closer the twin gets to live operations, the less forgiving the assurance problem becomes.

Fourth, the relationship between the digital twin and AI agents may not be cleanly separable. The paper assumes the twin can be assured separately from each AI agent, but acknowledges that operationalising a specific agent and twin together may require a separate assurance case. Emergent behaviour can arise from interaction. Anyone building multi-agent systems should underline that sentence, then underline it again in a less optimistic colour.

Fifth, synthetic data cannot be validated only by whether the LLM follows prompts. The prompts themselves must be calibrated against real-world distributions and expert judgement. A perfectly obedient generator can still generate the wrong world.

These boundaries do not weaken the paper. They prevent misuse. Assurance is not a magic stamp. It is an argument whose validity depends on scope.

The real contribution: assurance as a design interface

The most useful way to read this paper is not as an aviation paper about a digital twin. It is a paper about assurance as a design interface.

The TEA case forces technical teams, domain experts, regulators, and stakeholders to negotiate the same object: a structured map of claims and evidence. That map exposes where the system is mature, where it is approximate, where evidence is missing, and where assumptions are doing too much work.

For AI products, this is a better habit than the usual demo-first culture. Demo-first culture asks whether the system can perform impressively under selected conditions. Assurance-case culture asks what must be true for the system’s performance to count as evidence.

That difference matters more as AI systems become agents. A chatbot answer can be wrong in one place. An agentic system can be wrong through data ingestion, state representation, planning, action execution, logging, and feedback. The failure is not a point. It is a path.

Project Bluebird’s framework is valuable because it follows the path.

It starts with the data pipeline, where reality enters the system. It moves to virtual representation, where reality is compressed into software. It moves to trajectory prediction, where the future is estimated under uncertainty. It ends with agent interoperability, where AI systems act inside the simulated world and are judged by human and quantitative standards.

That category-by-category structure is the paper’s practical gift. It gives teams a way to avoid the childish question “Do we trust the AI?” and replace it with a better one:

Trust which claim, supported by which evidence, under which assumptions, for which use case?

Less catchy. Much harder to fake.

Cognaptus: Automate the Present, Incubate the Future.