The 99% Problem: When a Stroke Benchmark Looks Ready Before It Is

TL;DR for operators

A model reporting 99% accuracy creates an obvious decision pressure: should the team fund integration, start clinical workflow design, or treat the experiment as essentially solved?

This paper is a useful example of why that decision cannot be made from the headline number alone. Its main results table reports a Stacking Classifier at 99.81% accuracy, Random Forest at 99.52%, and Bagging at 99.45%.¹ But those results come from one public Kaggle dataset after the original 5,110 records—249 stroke-positive and 4,861 stroke-negative—were reduced to a balanced working dataset containing 249 observations in each class.

That does not make the comparison meaningless. It changes what the comparison can support. The paper provides evidence that, under its prepared benchmark, several ensemble methods outperform the simpler alternatives it evaluates. It does not establish that a 99% figure will survive on an independent hospital population, under the original disease prevalence, or inside a clinical decision process.

For operators, the practical lesson is governance: before a benchmark score triggers product investment, the team needs a reproducible record of how the data were sampled, transformed, split, evaluated, and reported.

A near-perfect score is an escalation trigger, not a deployment decision

Suppose a healthcare analytics team compares several predictive models and one configuration approaches perfect accuracy. The immediate temptation is to treat model selection as finished. The remaining work appears to be engineering.

Md Shahriar Sajid’s 2026 undergraduate thesis¹ gives exactly that kind of result. Its principal performance table places three ensemble approaches at the top: stacking at 99.81%, random forest at 99.52%, and bagging at 99.45%. Decision tree follows at 98.30%, while KNN and TabNet are reported at 96.72% and 96.22%. The feedforward network reaches 93.15%, SVM 88.24%, and logistic regression 78.84%.

Within that table, model choice clearly matters.

The highest-ranked method combines the outputs of several models and lets another model learn how to use those predictions. That approach is called stacking. The paper’s broader rationale is that different classifiers capture different patterns, so combining them can reduce dependence on the weaknesses of any single learner.

The operational question, however, is not only which model won. It is what kind of evidence produced the ranking.

The 99% figure is conditional on how the dataset was rebuilt

The original dataset is extremely uneven: only 249 of 5,110 records are stroke-positive, while 4,861 are stroke-negative.

The paper does not run its main comparison on that distribution unchanged. Instead, it deliberately reduces the majority class until only 249 non-stroke records remain. This produces a 498-record dataset with equal numbers of stroke and non-stroke observations. That procedure is random undersampling.

This matters because a classifier’s reported accuracy is always tied to the population on which it is evaluated. A metric calculated after changing a roughly 1-to-20 class distribution into a 1-to-1 distribution is not a context-free statement about clinical performance.

The study reports an 80:20 train-test split and 5-fold cross-validation, alongside accuracy, precision, recall, and F1. Those are reasonable ingredients for a prototype comparison. But the source package also notes that the sequencing of preprocessing, undersampling, splitting, and cross-validation is not described consistently enough to reconstruct the evaluation pipeline precisely.

That distinction changes the operator’s question from:

Which model reached 99%?

to:

Which model reached 99% under exactly which data-construction and validation procedure?

The second question is the one a governance process needs to answer before benchmark results travel into product or clinical discussions.

The ensembles win the paper’s comparison, but the benchmark is doing part of the work

The paper’s strongest direct evidence is comparative. Under its reported balanced-data setup, ensemble methods occupy the top positions in Table 4.

This is useful prototype evidence. If an analytics team is deciding which models deserve another round of controlled evaluation, the table provides a rational starting point.

It is weaker evidence for a decision such as integrating the system into a hospital workflow. A model can perform well on one prepared dataset yet behave differently when tested on patients collected elsewhere, under different prevalence, measurement practices, demographic composition, or missing-data patterns.

Testing on such an independent population is external validation. The thesis does not report it.

Inconsistent results are themselves a governance signal

A second issue is easier to overlook because it does not concern model architecture at all.

The thesis does not report one consistent set of headline accuracies. Table 4 gives stacking 99.81%, random forest 99.52%, and bagging 99.45%. Elsewhere, the abstract and concluding chapter give different figures for several models; stacking is reported at 99.52% in those sections, while bagging appears at 99.27%.

The problem is not deciding which number is more flattering. The problem is that a business process needs to know which experimental artifact is authoritative.

When benchmark results move from a notebook into management review, product planning, or clinical discussion, experiment tracking becomes part of model governance. The preprocessing configuration, random seed where relevant, data split, model version, metric calculation, and approved result table need to resolve to one reproducible record.

The thesis therefore illustrates a practical failure mode that extends well beyond healthcare: a technically promising experiment can become operationally ambiguous before anyone has deployed a model.

What the paper supports, what Cognaptus infers, and what remains open

The paper directly shows that multiple ensemble classifiers outperform several simpler and neural alternatives within its balanced Kaggle benchmark. Its main table reports the strongest accuracy for stacking, followed closely by random forest and bagging.

Cognaptus infers that the next rational business step is not deployment but controlled replication. A team considering a stroke-risk product could preserve the original class distribution, lock the preprocessing order, repeat the evaluation across multiple splits, measure calibration, and test the shortlisted models on an independent dataset before using benchmark leadership to justify workflow integration.

That inference affects a specific decision: whether a prototype metric is strong enough to release additional product and clinical-integration resources. Under the evidence available here, the metric supports further validation expenditure more strongly than deployment expenditure.

What remains uncertain is clinically decisive. The study does not establish performance on an independent hospital cohort, prospective patients, relevant subgroups, or a live workflow. It also does not report the calibration evidence needed to know whether predicted risks correspond to observed frequencies. The highest-performing stacking configuration is not described in enough detail for complete reproduction.

Prototype ranking and clinical evidence belong on different gates

The paper is most useful when treated as a model-selection exercise rather than a deployment proof.

Its reported results justify asking why ensemble methods perform so strongly under this prepared benchmark and whether that advantage survives stricter testing. They also show why benchmark governance cannot begin after model selection. Sampling, preprocessing, split design, metric definitions, and experiment tracking determine what a performance number actually means.

For a healthcare analytics team, a 99.81% result should therefore open the next gate rather than close the evaluation process.

The next gate is evidence: reproduce the pipeline, test on the original distribution, measure calibration and subgroup behavior, validate on independent clinical data, and subject the intended workflow to safety review. Only then can the organization decide whether an impressive prototype score has become evidence for operational use.

Cognaptus: Automate the Present, Incubate the Future.