TL;DR for operators
A transaction fails repeatedly on a weak network. To a fraud model, the retries, interruptions, and irregular timing can resemble suspicious activity. Yet the apparent risk signal may describe infrastructure quality rather than fraudulent intent. Better calibration or a higher confidence threshold can identify uncertain cases, but neither explains the source of uncertainty nor determines who should resolve it.
The paper proposes a three-tier routing policy. Routine cases remain automated. Cases shaped by noisy operating conditions go to a specialist who can interpret network and transaction context. Cases the model cannot confidently distinguish—or that cross regulatory red lines—go to a senior reviewer, trigger a request for more information, or are refused when the required authority is unavailable.
Human attention is limited, so the policy also changes escalation thresholds as review queues fill while preserving priority for fairness-sensitive cases. The operational question is therefore not merely whether to add human review. It is which uncertainty should trigger which reviewer, how scarce capacity should be allocated, and when the system should decline to decide.
In the modified PaySim evaluation, this hierarchical strategy reduced the reported rural–non-rural accuracy gap from 19.43 to 2.88 percentage points while routing 7.2% of test transactions to humans. The result supports testing uncertainty-specific, capacity-aware routing; it does not establish live-deployment performance or prove that structural inequality has been eliminated because the transactions and reviewers were simulated or modeled.
A retry can be a fraud signal or a network signal
Consider a customer whose payment repeatedly fails on an unstable connection. The transaction record may show retries, interruptions, irregular timing, and unusual velocity. Those features can resemble account takeover or automated fraud.
An autonomous model may classify the pattern correctly according to its training distribution while still making the wrong operational judgment. Its score describes similarity to past fraudulent behavior; it does not identify whether the current cause is malicious intent, weak infrastructure, or a novel condition outside the model’s experience.
This is where aggregate accuracy can conceal a deployment problem. The AI-only baseline in the paper achieves 97.19% overall accuracy, yet rural accuracy is 78.52% compared with 97.95% for non-rural transactions. The 19.43-point gap indicates that one operating environment bears a much larger share of the errors.
Raising the automation threshold would send more cases to review, but it would not determine who should review them or why. It could also consume analyst capacity on ordinary low-confidence cases while still missing the structurally exposed transactions that need priority.
The system separates noisy conditions from model ignorance
The paper’s main design choice is to classify uncertainty before assigning authority.
First, the random-forest model is calibrated with Platt scaling. Its reported weighted expected calibration error is 0.0083, indicating a small average gap between predicted confidence and observed accuracy across confidence bins in this test setting. A cost rule then sets the automation threshold at 0.90 because a missed fraud is assumed to cost ten times as much as a human escalation.
Calibration supports thresholding, but the system does not treat every case below the threshold as equivalent.
For infrastructure-related ambiguity, the paper averages predictive entropy across the forest’s 100 trees. Transactions above the 85th percentile are treated as high-noise cases. The interpretation is that the data-generating conditions are unstable even if the model itself is familiar with the general pattern. The paper labels this aleatoric uncertainty.
For deeper model ignorance, the system uses adaptive prediction sets with 90% coverage. When the prediction set contains more than one plausible class, the model is treated as unable to distinguish the outcome confidently. This epistemic uncertainty routes the case toward the senior red-line pathway.
The distinction changes the operating model:
| Case condition | Primary destination | Intended action |
|---|---|---|
| High confidence and no red line | Autonomous AI | Approve or reject automatically |
| Infrastructure-related ambiguity | Specialist analyst H1 | Interpret retries, timeouts, and contextual evidence |
| High-stakes model ignorance | Senior supervisor H2 | Review with greater authority, request information, or refuse |
| Senior capacity unavailable for a required case | Refusal pathway | Avoid an unsupported automated decision |
The contribution is not simply “add a human.” It maps different failure causes to different expertise, authority, and permissible actions.
The reported gains are concentrated where the baseline fails
The headline results are substantial within the paper’s held-out synthetic test set.
| Metric | AI alone | Hierarchical strategy | Change |
|---|---|---|---|
| System accuracy | 97.19% | 99.07% | +1.88 pp |
| Fraud recall | 66.39% | 91.18% | +24.79 pp |
| Rural accuracy | 78.52% | 96.30% | +17.78 pp |
| Non-rural accuracy | 97.95% | 99.18% | +1.23 pp |
| Rural–non-rural gap | 19.43 pp | 2.88 pp | −16.55 pp |
The overall accuracy increase is smaller than the recall and rural-accuracy gains because the baseline already performs well on the majority of cases. The routing policy adds most value in the subset where missed fraud and infrastructure-linked error are concentrated.
The paper also reports a positive complementarity gap of 1.880%, meaning that the modeled team has lower risk than the stronger standalone decision maker under the paper’s risk convention. Selective-risk analysis is lower for the collaboration policy around the chosen operating threshold. These measures support the claim that routing contributes more than merely replacing some model decisions with assumed-perfect human answers.
The resource profile is equally relevant. Of 25,000 test transactions, the autonomous AI handles 23,208, H1 handles 1,669, and H2 handles 123. Human review therefore covers 7.2% of volume, not all rural transactions and not every uncertain case.
That is the paper’s central operational proposition: large gains may be available from concentrating human attention on cases where context or authority can change the decision.
Fairness becomes a capacity-allocation rule
Selective review only works if the queue remains governable. The paper therefore adds mechanisms that are closer to operations research than to model training.
H1 attention is rationed with a dynamic shadow price. As the specialist queue approaches its 15% budget, the effective cost of escalation rises and the gate becomes stricter. This prevents a static threshold from overwhelming the review team during high-volume periods.
A rural-priority override bypasses that gate for fairness-sensitive cases. Without the override, transactions exposed to infrastructure-related proxy bias could be rationed away precisely because the queue is full. The policy therefore makes a deliberate choice: capacity efficiency does not have final authority over cases with elevated structural risk.
H2 has a 2% budget and a refusal rule. When a case requires senior authority but that authority is unavailable, the system can decline to decide rather than silently downgrade the case to ordinary automation. A 5% random urban audit is also included to maintain exposure outside the specialist pathway and reduce the risk that reviewers lose competence on routine or non-rural cases.
For a fraud operations team, the relevant design questions become concrete:
- Which signals indicate noisy infrastructure rather than suspicious intent?
- Which cases require contextual interpretation, and which require higher decision authority?
- Which groups receive priority when queues tighten?
- What decisions must be refused when the required reviewer is unavailable?
- Which segments should be audited to detect drift and preserve reviewer capability?
Cognaptus infers that these controls should be governed as a single routing policy. Calibration, fairness monitoring, queue thresholds, escalation authority, and refusal behavior will produce inconsistent outcomes if they are owned by separate teams without a shared decision specification.
The fairness gain is partly paid for in waiting time
The system’s mean wait rises from zero under AI-only processing to 3.08 minutes. The distribution is uneven: rural transactions wait 15 minutes on average, while urban transactions wait 2.6 minutes.
This is not a secondary service metric. The policy reduces false decisions for the group most exposed to infrastructure noise, but it also asks that group to wait much longer for resolution. A customer may prefer a delayed approval to an immediate false rejection, yet repeated delays can still affect merchant trust, liquidity, abandonment, and access to time-sensitive transactions.
A deployment team should therefore evaluate fairness across at least two linked outcomes: decision quality and service latency. A routing rule that narrows an accuracy gap while creating an unacceptable delay burden has shifted the form of disadvantage rather than fully resolved it.
The paper does not optimize this trade-off. It demonstrates one operating point. Live testing would need group-specific queue metrics, abandonment rates, analyst service times, escalation outcomes, and explicit service-level constraints. Additional reviewer capacity may improve latency, but it would change the economics that make the 7.2% review share attractive.
The evidence supports a design trial, not a deployment claim
Several boundaries materially limit interpretation.
The 100,000 transactions are a modified sample from PaySim, not records collected from Nigerian financial institutions. Urban and rural patterns, account tenure, the 8.21% fraud rate, and infrastructure effects are constructed for the experiment. External validity therefore remains untested.
Human expertise, fatigue, review costs, queue pressure, and audit effects are modeled assumptions. The study does not observe analysts working under live service conditions. It also reports no confidence intervals, repeated-run variability, or detailed hypothesis tests despite describing the improvement as statistically significant.
The mechanisms are evaluated together. There is no ablation separating the effects of calibration, entropy routing, adaptive prediction sets, shadow pricing, the rural override, refusal, or random auditing. The results show that the combined policy performs better in this simulation; they do not identify which component contributes how much.
Finally, the fairness analysis is primarily a binary rural-versus-non-rural accuracy comparison. Other forms of structural disadvantage, intersectional effects, and within-region variation are outside the reported evaluation.
Fraud automation needs a routing policy, not only a score
The paper changes the deployment decision from “How confident must the model be?” to “What kind of uncertainty is present, who has the authority to resolve it, and what happens when that authority is scarce?”
Within its synthetic setting, the answer is promising. The system automates 92.8% of transactions, recovers many fraud cases missed by the baseline, and sharply narrows the reported regional accuracy gap. It also exposes the cost of that improvement through asymmetric waiting time.
For fraud leaders, the next step is not to copy the reported thresholds. It is to test whether local transaction features encode infrastructure disadvantage, calibrate uncertainty signals on institutional data, measure reviewer complementarity rather than assume it, and define queue overrides and refusal rules before deployment pressure makes those choices implicitly.
Cognaptus: Automate the Present, Incubate the Future.