Structure, Stress, and Secrets: The Three Tests Production AI Keeps Pretending Are One

TL;DR for operators

Production AI is usually evaluated as though one good model score can certify the entire system. It cannot. A model can be efficient because the task was structured intelligently, appear reliable because the test users were unusually cooperative, and still expose sensitive information through the infrastructure that serves it.

Three recent papers line up into a useful production-readiness chain. The first shows how to encode stable domain rules so a learned policy does not waste inference time rediscovering them. The second shows why evaluation must vary the unstable part of the system: human communication, emotion, persistence, literacy, and willingness to cooperate. The third shows why deployment claims must be decomposed into precisely what is protected, what remains observable, and what an attacker can reconstruct from repeated exposure.

The operator’s framework is simple:

1. Structure the task: encode legitimate constraints, intermediate features, and action validity before spending compute on generic learning.
2. Stress the interaction: evaluate the same underlying cases through different user behaviours, not just polished benchmark prompts.
3. Name the secrets: separate query confidentiality, document privacy, linkability, and access-pattern leakage instead of calling the whole arrangement “secure.”

The broader conclusion is less convenient than “the model works,” but more useful: production readiness is not a property of the model. It is a property of the model, the evaluation distribution, and the deployment protocol taken together.

Production AI has three exams, not one

The current AI deployment cycle has an odd habit. Teams train a model, test it on a benchmark, wrap it in an API, and then discuss “readiness” as though those three steps measure the same thing. They do not.

A benchmark can establish that a system performs a task under specified conditions. It cannot establish that the conditions resemble real use. Encryption can protect a numerical value in transit or during computation. It cannot, by linguistic enthusiasm alone, protect the surrounding metadata, candidate identifiers, stored geometry, or repeated-release invariants. And a larger model can absorb some domain regularities, but that does not make it rational to force the model to rediscover rules the product team already knows.

The three papers considered here are not about one application. One studies reinforcement learning for Indian Rummy, one studies conversational medical AI, and one studies private dense retrieval. That variety is exactly why the cluster is useful. Together, the papers trace three adjacent questions in the lifecycle of a production system:

• What stable structure should be built into learning?
• What unstable variation must be introduced into evaluation?
• What residual information must be acknowledged after deployment?

The papers do not jointly prove a universal law of AI engineering. They do something more practical: they expose three places where teams commonly substitute a convenient proxy for the property they actually care about.

Test one: encode the stable structure before buying more compute

IRumAI tackles a difficult hidden-information game with an unusually practical design choice: it does not ask reinforcement learning to infer every rule-shaped feature from raw cards and sparse wins.¹ Indian Rummy requires players to assemble valid sequences and sets, satisfy declaration constraints, infer an opponent’s interests from visible draws and discards, and act under a very large combinatorial state space.

The paper’s solution is not “PPO, but with confidence.” It constructs a learning problem around the domain.

The observation includes ten card-level channels covering the player’s hand, visible history, wildcard status, seen cards, meld coverage, live outs, and partial melds, plus global structural features. The action space is masked so the policy cannot select phase-inappropriate actions, and declaration is exposed only when the hand is legally valid. The network itself has separate convolutional paths for the two core geometries of the game: horizontal rank adjacency for sequences and vertical suit aggregation for sets. A potential-based reward derived from deadwood reduction supplies intermediate learning signal while preserving the original optimal-policy set.

This is not merely feature engineering wearing a neural-network hat. It is a division of labour:

• Rules determine what is impossible. Action masking removes invalid choices.
• Domain analysis determines what is expensive to rediscover. Meld progress and live outs are surfaced explicitly.
• Architecture determines which relations should be easy to represent. Sequence and set patterns receive separate inductive biases.
• Learning determines strategy under uncertainty. The policy still decides how to trade off hand improvement, information exposure, and opponent blocking.

The experimental hierarchy is revealing. The learned policy achieves a 53.87% mean win rate against the strongest search-based opponent, which it did not face during reinforcement-learning training, while requiring a reported 0.33 milliseconds per action for the policy forward pass on a single ARM CPU core. The strongest search baseline takes roughly 2.4 seconds per action. The model is not merely faster because neural networks are fashionable; it is faster because the expensive combinatorial work has been converted into structured state representation and amortised policy inference.

The ablations sharpen the point. Replacing the dual-branch architecture with a vanilla multilayer perceptron costs roughly 8 to 11 percentage points across the evaluated opponents and pushes performance below 50% against the strongest baseline. Reward shaping adds a smaller but consistent improvement. Behaviour cloning mainly accelerates early learning; by later training, the cold-start run approaches similar performance. A more elaborate opponent-sampling curriculum produces mixed results because the opponent pool lacks the strategic diversity that would justify it.

In other words, the largest gain comes from matching the representation to the problem, not from adding another layer of training ceremony. The industry has somehow managed to make this sound radical.

The business interpretation

For production systems, the lesson is not to hard-code every business rule into a brittle expert system. It is to distinguish three categories that teams often blur:

This separation can reduce sample complexity, prevent nonsense actions, lower inference cost, and make failure analysis less theatrical. It also improves governance: when a rule is explicit, operators can inspect and change it without pretending that a new prompt constitutes a control system.

But IRumAI also marks the boundary of this success. Its world is simulated, rule-complete, two-player, and single-round. Its reinforcement-learning opponents come from a narrow heuristic family. The paper evaluates transfer to unseen heuristics, not to humans, and the ablations use one seed. The model demonstrates that domain structure can produce efficient strategic competence inside a controlled environment. It does not demonstrate that the resulting competence survives the full variety of real opponents.

That boundary leads directly to the second test.

Test two: vary the user, not just the case

The patient-centred conversational AI study begins where many benchmarks quietly stop: with the observation that users do not behave like evaluation prompts.² The researchers analysed 2,053 real patient-chatbot sessions. Only 14% reached a completed triage recommendation, 9% ended through an emergency flag, and 77% were abandoned before completion. Among sessions with at least two patient turns, 67% of patient responses contained fewer than six words. Emotional signals appeared in 37% of conversations, and non-standard communication features appeared in 79%.

That distribution is awkward for systems evaluated on articulate synthetic patients who answer every question, disclose the relevant facts, remain on topic, and politely wait for the conclusion. Those simulated users are not neutral. They are the best customers the model will ever have.

To examine the effect, the paper builds a modular patient simulator that separates four elements:

• clinical content,
• emotional state,
• conversational strategy,
• surface communication style.

The separation matters. It allows the researchers to hold the clinical facts constant while changing how the patient communicates them. The simulator supports parameters covering recall, oversharing, anxiety, scepticism, abandonment, English proficiency, health literacy, verbosity, punctuation, typos, and related behaviours. This is an evaluation instrument, not an attempt to declare that five personas capture humanity. The paper is commendably explicit about that limitation.

The simulator performs strongly on its internal checks. Across 950 paired comparisons, the activated behavioural parameter is identified with a macro concordance statistic of 0.894. In a clinical-fidelity evaluation, supported claims receive a mean accuracy rating of 3.94 out of 4, with no contradictions recorded. Human graders distinguish simulated from real conversations at 55% accuracy, close to chance, although one LLM judge detects simulation more reliably by noticing regularities that humans miss.

The crucial experiment applies five communication personas to 1,164 clinician-graded cases and evaluates four LLM-based triage systems. The clinical content remains the same; the presentation changes.

The resulting shifts are not cosmetic. For one evaluated model, the anxious persona produces an over-triage rate 13.5 percentage points higher than the dismissive persona. Under-triage moves in the opposite direction. The same ordering appears across the other models, though with smaller magnitudes. Discrimination remains broadly similar across personas, while calibration shifts. The systems continue to rank cases similarly, but alter how aggressively they map those cases to urgency levels depending on how the patient sounds.

This distinction is important. A model can retain decent discrimination and still make systematically different operational decisions because its threshold behaviour moves with communication style. A dashboard that reports only aggregate accuracy may not notice. A patient certainly might.

The business interpretation

The paper’s direct domain is healthcare, where under-triage can delay care and over-triage can waste scarce resources. The broader product lesson applies to any conversational system whose outcome depends on information elicited from a person.

The user is part of the input distribution. More precisely, the interaction protocol creates the input distribution.

A customer who gives a polished paragraph, a customer who answers “idk,” a customer who becomes frustrated after four questions, and a customer who overstates urgency may carry the same underlying need. They do not present the same machine-readable case. The model’s ability to extract, calibrate, and act on the need is therefore a property of the entire conversation, not merely of the final prompt.

This has at least four operational consequences.

First, completion is a performance metric. A system that is accurate after 17 turns but loses most users after one turn has not achieved 90% accuracy in production. It has achieved an attractive conditional result on the fraction of users willing and able to complete the process.

Second, interaction style can function like a hidden subgroup variable. Traditional fairness checks may stratify by age, sex, language, or geography. Communication behaviour can cut across all of them and may be harder to observe safely. It still affects outcomes.

Third, calibration needs distributional testing. Teams should not assume that one global confidence threshold remains appropriate across terse, anxious, sceptical, or low-context conversations. Any attempt to recalibrate by communication features must itself be governed carefully, because inferring user traits can create new risks.

Fourth, synthetic evaluation needs adversarial realism, not decorative diversity. Changing names and demographic labels while every simulated user remains cooperative does not produce a realistic test population. It produces a brochure.

The paper also resists overclaiming. It studies one triage task with a three-level outcome. It uses a limited set of personas and commercial models. The simulator is grounded in digitally connected users, which may underrepresent people on the other side of the digital divide. Realism optimisation introduces its own trade-off with strict parameter adherence. And because commercial models change, the results are a time-stamped snapshot, not a permanent ranking.

That caution is not a weakness. It is the correct posture for an evaluation layer whose purpose is to reveal where confidence should stop.

Test three: describe the leakage channel before describing the system as secure

SHARD moves the logic chain from evaluation into infrastructure.³ The setting is dense retrieval: documents are represented as embeddings, an approximate index retrieves candidates, and a reranking stage computes more precise similarity scores. The privacy problem is that embeddings preserve useful semantic structure precisely because they retain information about the source text. A compromised vector store may therefore reveal far more than a harmless numerical fingerprint.

The paper studies a hybrid construction that splits each centred embedding into two parts. A short public prefix supports coarse routing. The remaining residual is partitioned into cells, and each cell receives its own secret orthogonal transform. During reranking, the query residual is transformed with the corresponding cell key and evaluated through CKKS approximate homomorphic encryption. In exact arithmetic, the keys cancel inside the query-document inner product, so the corrected score preserves the raw document ranking up to a query-dependent constant.

That mechanism delivers a real utility result. Across ten BEIR and MIRACL encoder-dataset configurations, corrected full scoring reproduces raw ranking behaviour, with the remaining small loss attributed to shortlist recall rather than score distortion. In 315 measured CKKS trials, the implementation reports no top-1 flips and full top-10 overlap. The paper also shows that block packing reduces query upload, although the tested implementation becomes slower because it still returns a separate encrypted response per candidate.

So far, this sounds like the familiar privacy pitch: keep retrieval quality, add encryption, declare victory, choose a tasteful lock icon.

The paper refuses the shortcut.

It separates three privacy notions:

1. Query-value confidentiality: CKKS hides the numerical query vector and score values from an honest-but-curious server without the secret key.
2. Document privacy: the stored representation should resist recovery or re-identification of document content.
3. Access-pattern privacy: the server should not learn which candidates, cells, or document identifiers are repeatedly accessed.

SHARD helps with the first and complicates attacks on part of the second. It does not solve the third. It also does not provide differential privacy, unlinkability, a cancellable template, or a cryptographically private document store.

The reason is structural. Independent cell keys compartmentalise alignment evidence: known document-embedding pairs are spread across cells, slowing a diffuse attacker’s accumulation of anchors. But the protection is gradual, not thresholded. Minimum-norm alignment can recover useful signal from a rank-deficient set of known pairs, well before a full cell key is mathematically identified. A targeted attacker can concentrate anchors in one cell.

Re-keying does not erase stable invariants either. The public prefix remains unchanged. Orthogonal transforms preserve residual norms. Shared cell keys preserve within-cell Gram geometry. Those features can link records across releases even when row order and identifiers change. Micro-keys remove shared residual comparisons, but leave the prefix and norm channels. The paper’s conclusion is deliberately narrow: SHARD is a rank-preserving alignment-compartmentalisation mechanism with explicit leakage, not an all-purpose privacy guarantee.

The business interpretation

The immediate lesson for RAG and enterprise search teams is to replace security adjectives with a leakage ledger.

A deployment review should state, channel by channel:

This is more useful than asking whether the vector database is “encrypted.” Encrypted where? Against whom? Which values? Under what key ownership? What remains visible before and after the encrypted computation? How does the system behave after repeated queries or re-indexing?

The paper also supplies a broader governance principle: a mechanism’s measured success does not enlarge its security claim. The fact that CKKS preserves ranking does not imply document confidentiality. The fact that an unknown rotation reduces one off-the-shelf inversion model’s performance does not imply resistance to adapted attacks. The fact that re-keyed coordinates look different does not imply unlinkability.

A serious security description is therefore a set of bounded statements, not a product adjective.

The combined logic: structure, stress, secrets

Placed together, the three papers form a production chain rather than a collection of case studies.

The common error is local optimisation. IRumAI could be judged solely on win rate. The triage systems could be judged solely on average agreement with clinicians. SHARD could be judged solely on ranking fidelity and encrypted arithmetic. Each local metric is legitimate. None is sufficient for the system-level claim people are tempted to attach to it.

A useful operator heuristic is:

This is not an empirical formula from the papers. It is a management interpretation of their combined logic. The minimum matters because the weakest layer can invalidate the commercial value created by the others.

A fast policy that fails against real users is not ready. A conversational model that performs well for articulate users but loses terse users is not broadly reliable. A retrieval system that preserves accuracy and encrypts scores but leaks persistent document identity is not private in the way procurement may assume.

An operator’s three-gate review

The papers suggest a practical review process for teams moving an AI feature toward production.

Gate 1: Structure review

Before training or fine-tuning, ask:

• Which actions are always invalid and should be masked or blocked?
• Which intermediate quantities are expensive but deterministic enough to compute directly?
• Which relationships should the architecture represent naturally?
• Which reward components reflect the true business objective, and which are merely convenient proxies?
• Which domain rules may change and therefore need configurable rather than hard-coded implementation?

Measure not just model quality, but also invalid-action rate, sample efficiency, inference latency, rule coverage, and sensitivity to removing each structural component.

The IRumAI result is especially relevant here: its architecture ablation matters more than its curriculum embellishment. That is a warning against solving representation problems with training complexity.

Gate 2: Interaction stress review

Before launch, hold the underlying case constant and vary the path by which information reaches the system.

Test terse users, verbose users, emotional users, sceptical users, users who omit facts, users who answer out of order, and users who abandon early. For non-conversational systems, the equivalents may be sparse records, delayed events, inconsistent formatting, sensor dropout, or adversarial workflow behaviour.

Track:

• task performance by interaction persona,
• calibration and threshold movement,
• completion and abandonment,
• number of turns or corrections required,
• escalation rates,
• recovery after missing or contradictory information.

Do not compress these into one average unless the business is prepared to serve only the average user, a demographic with a surprisingly small mailing list.

Gate 3: Leakage review

Before calling the deployment secure, write the threat model in operational language.

Identify:

• who holds each key,
• what the server stores in plaintext,
• what is encrypted during computation,
• what metadata is visible,
• what repeated queries reveal,
• what changes across releases,
• what remains invariant,
• what a known-pair or overlapping-corpus attacker can exploit.

Then map each product claim to evidence. “Query values are encrypted under an honest-but-curious server model” is a defensible statement. “The document store is private” may not be. Precision is not pedantry here. It is the difference between a bounded engineering claim and an expensive future meeting.

Where the analogy stops

The three papers operate in very different domains, and the chain should not erase those differences.

Indian Rummy provides complete rules and a simulator capable of generating hundreds of millions of training steps. Human communication does not offer comparable completeness. Its relevant dimensions are socially and culturally contingent, partly unobserved, and expensive to validate. Dense-retrieval security adds an adversary who deliberately exploits invariants; a patient or card-game opponent is not necessarily behaving under the same threat model.

The cluster therefore does not imply that every system should use the same technical method. A patient simulator should not be treated like an action mask. A cryptographic protocol should not be evaluated like a persona. The shared lesson concerns where explicit structure belongs:

• stable rules belong in system design,
• variable human behaviour belongs in evaluation distributions,
• adversarial observability belongs in deployment threat models.

Confusing those categories produces familiar mistakes. Teams learn what they could have encoded, benchmark what they should have varied, and hide what they should have disclosed.

What leaders should ask for next

For executives and product owners, the most useful deliverable is not another model leaderboard. It is an evidence map connecting a commercial claim to the layer that supports it.

This reframes AI governance away from ceremonial checklists. The point is not to slow deployment until uncertainty disappears. It will not. The point is to prevent one layer’s success from being used as evidence for another layer’s unresolved risk.

IRumAI earns confidence in efficient policy execution under a carefully structured game environment. The patient-simulation study earns concern about evaluation distributions built from idealised users. SHARD earns a precise claim about rank-preserving compartmentalisation and an equally precise list of what remains exposed. Each paper is strongest where it resists the temptation to generalise beyond its evidence.

Production teams should do the same.

The model is only the middle

The most consequential AI failures increasingly come from the spaces around the model.

Upstream, the task is poorly structured, so learning spends compute on avoidable ambiguity. At evaluation time, synthetic users behave better than real customers, so reported quality describes a preferred interaction rather than the deployed population. Downstream, a protection mechanism secures one numerical channel while leaving identifiers, geometry, or behaviour visible, so “private” becomes an optimistic summary of a much narrower result.

The three-paper chain offers a cleaner operating doctrine:

• encode what is genuinely stable,
• vary what will be unstable,
• disclose what remains observable.

That will not produce a single triumphant readiness score. It will produce something better: a system whose claims are attached to evidence, whose limitations are legible, and whose operators know which layer needs work next.

A mildly disappointing outcome for marketing, perhaps. A considerable improvement for everyone else.

Cognaptus: Automate the Present, Incubate the Future.