Course Correction: Ask What Learners Use, Not What Courses They Took

TL;DR for operators

A training intake form usually asks whether someone has completed an AI course or workshop. It is tidy, auditable, and apparently not very informative.

In a study of 93 bioscience graduate and postdoctoral trainees, reported LLM usage frequency differentiated all five baseline AI-perception outcomes after correction for multiple comparisons. Self-rated familiarity differentiated three. Prior AI education differentiated none.¹

The strongest operational signal was therefore not what learners had formally studied, but how often they said they used LLMs. That does not mean usage measures competence. It means usage was more consistently associated with learners’ trust, confidence, concern about over-reliance, and interest in training.

For course designers, the sensible intake sequence is:

1. Ask how frequently the learner uses LLMs.
2. Use self-rated familiarity as a secondary check.
3. Treat prior coursework as contextual information, not a sufficient routing rule.
4. Give inexperienced users orientation and relevance-building support.
5. Give highly engaged users calibration exercises focused on failure modes, verification, and over-reliance.

The boundary is substantial. The study is cross-sectional, single-institutional, and based entirely on single-item self-reports. It measures perception differences, not objective AI literacy, judgment quality, or learning outcomes. The result supports inexpensive segmentation. It does not confer psychometric sainthood on a five-option usage question.

The credential checkbox loses its first contest

Every training program loves a credential checkbox.

Have you attended an AI workshop? Completed a course? Earned a certificate? Excellent. The database now contains a clean binary field, and everyone can return to pretending that exposure and readiness are interchangeable.

The paper tests that assumption against two alternatives: how frequently learners report using LLMs and how familiar they believe themselves to be with them. All three signals are inexpensive to collect. All three sound plausible. Only one consistently separates the baseline perception patterns that matter for instructional design.

That is what makes this study useful. It is not proposing a sophisticated learner model with clickstream telemetry, latent-state estimation, or a neural network wearing a pedagogical hat. It asks a narrower and more practical question:

Which simple intake item tells an instructor the most about how learners enter an AI ethics course?

The comparison produces a clear ranking.

The ranking is not a verdict on the value of education. It is a verdict on the information contained in these particular intake measures, for this particular cohort, before instruction began.

That distinction will matter later.

One cohort, three signals, five different perceptions

The study draws on an anonymous pre-course survey administered to 93 bioscience graduate and postdoctoral trainees enrolled in a required Responsible Conduct of Research course at UCLA. Two respondents left all five focal outcomes blank, leaving 91 valid cases for the association tests.

The cohort was predominantly composed of early-stage doctoral students and covered more than ten bioscience subdisciplines. ChatGPT dominated reported tool use, while common applications included searching for scientific information, summarizing literature, and understanding complex concepts.

The researchers compared three candidate intake features:

• Usage frequency, from “Never” to “Daily”
• Self-rated LLM familiarity, from “Not at all familiar” to “Very familiar”
• Prior AI education, categorized as formal, informal, or none

These were tested against five separate perception items:

1. Trust in LLM accuracy for general scientific information
2. Confidence in distinguishing factual from incorrect LLM output
3. Trust in LLMs for complex ethical or scientific questions
4. Concern that over-reliance could impair critical thinking
5. Interest in formal LLM training

The paper deliberately does not combine these items into one composite score. That is methodologically sensible because they are not interchangeable manifestations of a single obvious trait.

Someone can trust LLMs for routine scientific information while distrusting them on complex ethical issues. They can feel capable of spotting errors while remaining concerned about long-term over-reliance. They can also distrust the tools and still want training. Human attitudes continue to resist the convenience of a single dashboard number.

Because the responses are ordinal, the authors use Spearman correlations for usage frequency and familiarity. Prior education is tested with Kruskal–Wallis comparisons. Holm correction is applied across the five outcomes within each candidate-signal family.

That correction matters. Without it, a study testing several outcomes can collect chance findings with the enthusiasm of a child collecting stickers. The paper’s central ranking is based on the associations that remained after controlling the family-wise error rate.

Usage frequency separates every measured perception

Reported usage frequency was associated with all five outcomes after Holm correction.

The largest association was with trust in LLMs for complex tasks, at $\rho=.409$. Accuracy trust followed at $\rho=.370$, while concern about critical-thinking impairment moved in the opposite direction at $\rho=-.346$.

These are meaningful associations, but they are not remotely deterministic. Usage frequency is not a secret master variable from which an instructor can reconstruct the learner’s intellectual soul. It is simply the strongest of three low-cost signals tested here.

The group medians show why the authors describe part of the pattern as threshold-like rather than smoothly progressive.

For accuracy trust, the median rises from 2 among never-users to 3 among rare and occasional users, then to 4 among frequent and daily users. For training interest, the median jumps from 2 among never-users to 4 in every other usage group.

That pattern suggests an important distinction between non-use and some sustained engagement. Once learners have begun using LLMs, several attitudes shift sharply. Additional increases in frequency do not always produce equally clear additional movement.

The same shape does not appear everywhere. Trust in complex tasks increases more gradually, from a median of 1.5 among never-users to 2.5 among daily users. Concern about critical-thinking risk remains at a median of 5 from never through frequent use, then falls to 4 among daily users.

So there is no universal “used it once, transformed forever” threshold. The lower-end distinction is strongest for training interest and general accuracy trust. The critical-thinking pattern appears concentrated among the heaviest users.

This matters operationally because a single segmentation rule will not explain every attitude. A never-user may need help understanding relevance and basic operation. A daily user may require less introduction but more deliberate scrutiny of where confidence has become habitual.

Familiarity helps, but its scale runs out of room

Self-rated familiarity performed reasonably well, just not as consistently.

It remained associated after correction with:

• Distinguishing capability: $\rho=.294$, $p_{\text{Holm}}=.023$
• Complex-task trust: $\rho=.278$, $p_{\text{Holm}}=.031$
• Critical-thinking risk: $\rho=-.254$, $p_{\text{Holm}}=.045$

Its association with accuracy trust was positive but did not survive correction, with $p_{\text{Holm}}=.069$. Its association with training interest was essentially absent, at $\rho=.042$.

The tempting interpretation is that familiarity is simply a weaker psychological cousin of usage. The sample distribution suggests a more specific explanation.

Nearly four-fifths of the full cohort described themselves as either “Somewhat familiar” or “Very familiar.” Only one respondent selected “Not at all familiar.” Among the 91 valid cases, 47 were somewhat familiar and 25 were very familiar.

The scale was therefore crowded near the top. Once most respondents place themselves in two neighboring categories, the measure has limited room to distinguish them. The medians for somewhat- and very-familiar learners were frequently identical.

This does not make familiarity useless. It changes its role.

Usage frequency asks about reported behavior. Familiarity asks for a self-assessment of one’s relationship with the technology. When the two disagree, the discrepancy may itself be informative.

A learner might report frequent use but only moderate familiarity, perhaps indicating narrow or repetitive use. Another might claim high familiarity despite limited recent use, perhaps because of prior technical exposure—or because self-confidence has once again declined to wait for evidence.

The paper does not test these mismatch profiles, so they should not be treated as validated learner types. But the comparison supports a sensible intake design: use frequency first, then familiarity as a secondary check rather than an alternative master score.

Prior education loses the comparison, not the argument

Prior AI education showed no significant association with any of the five outcomes after Holm correction.

The first analysis divided learners into formal, informal, and no prior education. The authors then repeated the comparison after collapsing the categories into any prior education versus none.

The conclusion remained unchanged.

Under the two-group coding, the minimum corrected $p$-value was .524, and all absolute Cliff’s $\delta$ effect sizes were below .19. The largest median differences appeared for accuracy trust and critical-thinking risk, but neither was statistically significant.

This second coding is best understood as a robustness or sensitivity test. It asks whether the null result is merely an artifact of splitting prior education into too many categories. Combining formal and informal education increases the size of the “any prior” group and gives the comparison more statistical stability.

The same null result survives.

That supports a limited conclusion: the null is not obviously caused by the three-way coding choice.

It does not support the grander conclusion that AI education has no effect, courses are useless, and everyone should simply be handed a chatbot until wisdom emerges.

The education measure is coarse. It combines prior training in AI, computational biology, or bioinformatics and distinguishes only formal, informal, or none. It does not measure:

• What was taught
• How deeply it was taught
• How recently it was taught
• Whether the training covered generative AI
• Whether it included ethics, verification, or practical use
• Whether the learner retained or applied anything

A two-hour workshop and a semester-long technical course can therefore occupy neighboring administrative boxes. This is efficient recordkeeping but fairly heroic measurement.

The more defensible interpretation is that course-attendance history alone does not contain enough resolution to separate these baseline perceptions. Education may still matter. This particular label does not describe it well enough.

That is still a useful result. Many organizations route employees into training based on completion records because those records already exist. The paper shows why convenience should not be confused with predictive value.

The visual appendix summarizes the comparison; it does not add a second thesis

The paper’s main evidence is contained in the three statistical tables comparing the candidate signals. Figure 1 provides descriptive context by showing the cohort’s overall response profile. It is not a test of which intake variable performs best.

Figure 2, placed in the appendix, visualizes the group medians across usage, familiarity, and education. Its purpose is to make the comparative pattern easier to see:

• Usage shows the clearest separation.
• Familiarity shows weaker, lower-end separation.
• Prior education shows little consistent movement.
• The proposed lower-end threshold is not uniform across outcomes.

The figure is supporting visualization, not an independent analysis and certainly not a license to infer a perfect learner taxonomy from five colored dots.

The paper’s evidentiary pieces serve distinct roles:

Keeping those roles straight prevents the article from inflating an exploratory intake study into a completed adaptive-learning system. The paper itself is careful on this point. Here, “adaptive” means the possibility of intake-level grouping. It does not mean that the researchers built differentiated lessons, adaptive sequencing, personalized feedback, or an intelligent tutor.

The adaptive curriculum remains prospective. The intake comparison is the work actually completed.

More use brings more trust—and less worry

The most interesting result is not merely that usage frequency wins. It is the direction in which several perceptions move.

More frequent users report:

• Greater trust in general scientific accuracy
• Greater trust in complex-task performance
• Greater confidence in their ability to distinguish correct from incorrect output
• Less concern that over-reliance may impair critical thinking

That cluster can be read in two very different ways.

The optimistic reading is calibration through experience. Repeated users may have learned where LLMs perform well, developed practical verification habits, and become appropriately less anxious about risks they know how to manage.

The less comfortable reading is confidence through habituation. Repeated exposure may normalize reliance, increase subjective fluency, and reduce sensitivity to failures without improving the ability to detect them.

The study cannot distinguish between these mechanisms.

It does not test participants’ ability to identify fabricated citations, detect subtle scientific errors, challenge misleading reasoning, or decide when an LLM should not be used. “Distinguishing capability” is self-rated. The research therefore measures perceived evaluation skill, not demonstrated evaluation skill.

That is the central calibration boundary.

A learner who uses an LLM daily and feels highly capable might be genuinely proficient. The same response profile might also describe someone who has become extremely efficient at accepting articulate nonsense.

For training design, uncertainty about the mechanism is not a reason to ignore the result. It changes what the result is good for.

High engagement should not trigger automatic exemption from introductory AI ethics content. It should trigger a different form of instruction:

• Less time explaining that LLMs exist
• More time testing confidence under adversarial examples
• Less generic warning about hallucinations
• More practice identifying plausible but consequential errors
• Less emphasis on tool discovery
• More emphasis on verification, disclosure, and stopping rules

The useful business inference is not “experienced users need less training.” It is “experienced users may need different training.”

Conveniently, that is what adaptive instruction was supposed to mean before it became a synonym for adding a recommendation engine.

A practical two-signal intake model

The paper supports a lightweight intake process rather than an elaborate diagnostic instrument.

A sensible implementation could begin with two questions:

1. How often do you use LLMs?
2. How familiar do you consider yourself with their capabilities and limitations?

Prior education can remain on the form, but it should provide context rather than determine the route by itself.

An illustrative routing framework might look like this:

This framework is a Cognaptus inference, not an intervention tested by the authors.

The study does not show which lesson should be assigned to which group, how many groups are optimal, or whether routing improves retention, transfer, ethical judgment, or research behavior. Those questions require an experiment in which learners receive differentiated instruction and their outcomes are compared.

Still, the intake design has several practical advantages.

First, the questions are cheap. No platform integration is required, and no one needs to negotiate access to personal chat histories—a sentence that should not need writing, but enterprise AI has developed ambitions.

Second, the model is explainable. Learners and instructors can understand why a particular instructional route was selected.

Third, the segmentation can be treated as provisional. A brief diagnostic task can override the survey result when observed performance contradicts self-report.

That last step is essential. A behavioral intake signal is useful precisely because it helps decide where to look next. It should not become a permanent label.

The best signal may partly be the best-measured signal

There is another reason to resist overly theoretical conclusions from the ranking.

The three candidate variables do not have equal measurement resolution.

Usage frequency has five ordered levels and is reasonably distributed across them. In the valid sample, the groups contained 10 never-users, 13 rare users, 21 occasional users, 27 frequent users, and 20 daily users.

Familiarity also has five levels, but responses are concentrated near the top, including only one person in the lowest category.

Prior education has three broad categories and combines forms of training that may differ considerably in content and depth.

Usage frequency may therefore perform better partly because it contains more usable variation. It asks about a recurring behavior on a relatively balanced scale. The education variable asks respondents to compress a potentially complex history into a blunt category.

This does not invalidate the result. Operational signals compete as measured, not as idealized constructs floating above the survey instrument.

But it changes the theoretical claim.

The paper demonstrates that a five-level self-reported usage item is more informative than the particular familiarity and education items tested. It does not establish that behavioral engagement will always outperform a carefully designed measure of prior learning.

A future comparison might include:

• Number and duration of completed AI courses
• Recency of training
• Training content
• Assessed proficiency
• Observed LLM activity
• Diversity of tasks performed
• Frequency of verification behavior
• Performance on a short error-detection exercise

Usage frequency would then face a considerably less sleepy field of competitors.

What the study directly shows, what operators can infer, and what remains unknown

The paper is strongest when treated as a decision aid rather than a causal theory.

The distinction protects both sides of the business interpretation.

Without inference, the paper remains an interesting survey table with no operational consequence. Without boundaries, it becomes a readiness classifier assembled from self-report and optimism.

Neither extreme is particularly useful.

The boundary is perception, not proficiency

Several limitations materially affect how the result should be used.

The design is cross-sectional

The survey captures usage and perceptions at the same pre-instruction moment. It cannot establish direction.

Usage may alter trust. Trust may encourage usage. Both may arise from disciplinary norms, personality, workflow demands, or access to tools.

The observed correlations provide segmentation information without identifying the mechanism that created the segments.

Every focal measure is self-reported

Reported usage is not an activity log. Familiarity is not a skills assessment. Confidence in distinguishing errors is not demonstrated error detection.

Because predictors and outcomes use the same survey method, common response tendencies may contribute to the associations. Some respondents may generally choose stronger agreement categories or portray themselves as highly engaged and capable.

The familiarity scale has a fragile lower end

Only one respondent selected “Not at all familiar.” Any apparent difference involving that group is descriptive rather than stable. The stronger claim concerns the overall rank of familiarity as a secondary signal, not the precise attitude of a one-person subgroup.

Prior education is measured coarsely

The null result survives a simpler two-group coding, which strengthens confidence that it is not merely a category-splitting artifact. But the underlying item still does not capture training quality, duration, recency, or content.

The finding should weaken reliance on attendance labels, not weaken investment in well-designed education.

The sample is narrow

The participants came from one institution, were largely early-stage doctoral trainees, and worked within bioscience-related fields. Corporate employees, undergraduates, software engineers, clinicians, and senior researchers may display different relationships between use, familiarity, education, and trust.

No adaptive intervention was tested

The study evaluates candidate intake signals. It does not assign learners to differentiated modules and measure the consequences.

The key business proposition—better routing produces better learning—remains untested.

For deployment, the proper sequence is therefore:

The paper addresses the first arrow. Organizations still have to build and validate the rest.

Ask about behavior, then verify capability

The most useful result in this paper is not that one survey item achieved five corrected associations.

It is that three plausible administrative signals were forced into the same comparison.

Usage frequency won. Familiarity provided partial additional information. Prior education, as measured, contributed little separation.

That ranking offers a practical correction to how training programs often think about readiness. Course records describe what an institution has delivered. Usage describes what a learner reports doing. Familiarity describes what the learner believes that experience amounts to.

None measures competence directly.

A sensible AI ethics program should therefore ask about behavior first, use self-perception second, and verify capability before making consequential instructional decisions. Beginners may need orientation and a reason to care. Experienced users may need fewer definitions and considerably more opportunities to discover that confidence is not a quality-control system.

The business value is modest but real: a short intake survey can support cheaper, more relevant routing than a credential checkbox alone.

Just do not call the routing score “AI literacy” and wander off. The evidence has not been that generous.

Cognaptus: Automate the Present, Incubate the Future.