A CAPTCHA Pass Is Not a Human Response

TL;DR for operators

Low-cost autonomous agents can complete paid online surveys without recurring model-access fees, and conventional bot screening does not reliably identify them. In a controlled comparison of nine agent configurations, three fully open systems completed the survey, never triggered paste detection, and consistently failed a particular hidden-instruction check. Commercial agents exhibited different detection failures.¹

For survey platforms, market research firms, and purchasers of human-generated data, the implication is that neither automation cost nor successful browser screening provides dependable evidence of respondent authenticity. The study also finds that agents’ written answers differ substantially from human responses at the group level, even when their interaction behavior appears relatively human-like. This supports evaluating identity verification, procedural safeguards, and semantic analysis together. It does not establish that such a combined system will reliably identify individual automated submissions.

Completing the questionnaire is no longer the expensive part

A research platform pays participants to complete a questionnaire, applies quality checks, and accepts responses that appear legitimate. The resulting dataset is valuable because it is supposed to represent human beliefs, experiences, and judgments.

That assumption becomes fragile when a participant can delegate the entire questionnaire to an autonomous browser agent. The submitted form may be complete, internally coherent, and procedurally unremarkable while containing answers generated by a language model. This contamination of human research data with model-generated responses is often called LLM pollution.

Deployment economics have provided one potential constraint. If autonomous participation requires expensive commercial software, specialized technical knowledge, or substantial intervention, operating agents may be less attractive than recruiting genuine respondents.

Rilla and colleagues examine how much of that constraint remains when open-weight language models can be combined with readily available browser automation frameworks.

Their evaluation includes 360 autonomous survey completions, divided equally among nine configurations: three fully open, two mixed, and four closed commercial systems. The agents completed the same survey concerning attitudes and associations related to artificial intelligence. The researchers compared their behavior and responses with data from 3,242 U.S. participants recruited through Prolific.

The fully open configurations paired Devstral Small 2 24B, Ministral 3 14B, and Qwen 3 Coder 30B with Nanobrowser, an open-source browser-agent framework. All three completed the questionnaire without recurring API or subscription charges.

That does not mean their deployment was free. The local systems used substantial hardware, including an NVIDIA RTX 5090 GPU, and still required electricity and technical setup. The reported cost comparison excludes hardware acquisition and setup labor.

Their execution speeds also differed from commercial alternatives. Browser Use had a median completion time of 69 seconds, compared with 305 to 626.5 seconds across the three fully open configurations.

Nevertheless, the evidence changes the economic premise. A functional autonomous survey participant no longer necessarily requires recurring payments to a proprietary agent provider.

Cognaptus interpretation: For a survey platform assessing synthetic-participation risk, commercial agent pricing is becoming a weaker proxy for the cost of automation. The relevant cost assessment must include throughput, hardware utilization, setup, and the likelihood of detection. The paper demonstrates technical feasibility and low recurring model-access costs, not profitability at scale.

Different agents fail different detection checks

If inexpensive agents can reach the submission stage, the next question is whether existing safeguards can identify them.

The researchers instrumented the survey with sixteen binary detection checks, alongside response timing and browser-interaction measurements. These included reCAPTCHA, Cloudflare, hidden form elements, prohibited browser actions, AI-use disclosure, and deliberately embedded instructions.

The results reveal sharply different failure profiles.

Source: Table 1, 40 runs per configuration. Percentages describe failures of particular checks, not overall agent-detection accuracy.

The clipped jailbreak honeypot deserves particular attention. It placed an instruction within survey content in a way intended to expose automated systems that process hidden page elements. Compliance with such an instruction becomes a possible signal that the respondent is an agent rather than an ordinary human.

All five open and mixed configurations failed this check in every run, despite receiving task instructions to ignore embedded manipulations.

These configurations used the browser’s document object model (DOM) rather than live screenshots. Their access to underlying page content offers a plausible explanation for their exposure to hidden instructions. Commercial configurations could use rendered visual information and showed different behavior.

That architectural explanation is suggestive, not causally established: model selection, framework design, and visual capabilities varied together.

Two additional findings reinforce the danger of relying on a single safeguard. Cloudflare detected none of the 360 main-study runs, and all three prompt-leaking honeypots also recorded zero failures. An excluded pilot agent, Skyvern, did fail Cloudflare, so the result should not be generalized to every possible system.

The operational difficulty is that a check’s effectiveness depends on the agent interacting with it. A safeguard that consistently identifies one implementation may contribute almost nothing against another.

For research platforms, detection coverage must therefore be evaluated against multiple agent configurations rather than inferred from a successful test against one browser bot.

Human-like browser behavior does not guarantee human-like answers

Procedural screening examines how a submission arrives. Survey quality ultimately depends on what the submitted responses represent.

The researchers tested this distinction by comparing agent responses with the human reference sample across three types of information: open-text semantics, behavioral measures, and self-reported attributes.

For written answers, they generated 1,024-dimensional text embeddings and measured the standardized Euclidean distance between human and synthetic group averages. The calculation adjusts differences along each dimension by the pooled variance of that dimension.

A larger distance indicates greater separation between the groups’ average responses. It is not a probability that a particular answer was generated by AI.

The reported median semantic distances were:

Source: Figure 1. These are descriptive, group-level distances, not individual classification accuracies.

Open-text measures produced substantially greater standardized separation than the evaluated behavioral and self-report measures. The fully open configurations showed the greatest semantic divergence, even though they generally resembled human response timings more closely than larger commercial agents.

That difference matters for quality assurance. An automated respondent can reproduce plausible interaction timing without reproducing the distribution of associations, explanations, and judgments supplied by human participants.

Self-reporting provides little reassurance. Gemini 2.5 Flash disclosed AI use in 40% of runs, while other configurations rarely or never did so.

The semantic result also has an important measurement boundary. Distances calculated across high-dimensional embeddings aggregate differences over many features, and their absolute magnitude cannot be interpreted like a single-variable effect size. More importantly, separation between group centroids does not establish that individual human and synthetic responses can be classified with acceptable error rates.

The study identifies a promising source of additional detection information. It does not deliver a validated response-level detector.

Verification should cover access, behavior, and content

For organizations purchasing human-generated research data, the immediate question is how to allocate verification resources when conventional safeguards provide incomplete coverage.

The study supports evaluating three complementary control functions.

Participant and account verification addresses who is authorized to supply a response. Stronger identity and account safeguards may increase the difficulty of deploying multiple synthetic respondents, although their effectiveness and potential friction for legitimate participants require separate assessment.

Procedural instrumentation examines how responses are produced. Hidden elements, interaction restrictions, browser traces, and challenge systems may each expose particular agent behaviors. The study shows why these signals should be assessed as complementary rather than individually decisive.

Semantic assessment examines whether the content of submitted responses is consistent with an appropriate human reference population. Open-ended questions can provide information unavailable from clicks and completion times, but an operational system would need independently tested classification rules, appropriate reference samples, and measured false-positive rates before using such signals to reject participants.

This distinction matters for procurement. A survey supplier’s claim that its platform uses CAPTCHA or automated fraud checks does not establish that its delivered observations are genuine human responses.

A stronger assurance process would ask which agent configurations have been evaluated, how controls are combined, and what evidence supports the resulting acceptance decisions.

That is an implication for system design, not an experimentally demonstrated mitigation architecture. The researchers evaluated individual checks and response distributions; they did not test a fully integrated verification system.

What the results do not establish

The study provides controlled evidence about nine configurations completing one AI-related questionnaire. Its conclusions may not transfer directly to other survey topics, interface designs, languages, or future agent versions.

The agents were not fine-tuned on human responses or systematically optimized against the detection battery. More deliberate adversarial adaptation could change both procedural failure rates and semantic similarity.

The comparisons also cannot isolate the causal effect of model openness. The open, mixed, and closed categories differ in multiple architectural characteristics. Likewise, the human reference data were collected before the autonomous evaluations, and semantic distances depend on the selected respondents, embedding representation, and standardization method.

Finally, the research does not measure how frequently autonomous survey participation occurs on commercial platforms or whether operating these agents at scale is financially profitable.

These boundaries separate the demonstrated capability from the scale of the associated business threat.

Respondent authenticity requires more than successful screening

The central finding is that inexpensive autonomous participation, procedural detectability, and human-like response content are distinct properties.

Fully open agents can complete surveys without recurring model-access fees. Different agent architectures expose different weaknesses in conventional safeguards. And agents that resemble human respondents in browser behavior can remain substantially different in what they write.

For platforms and data purchasers, these findings justify testing layered verification rather than treating any single successful check as proof of authenticity. The next evidentiary requirement is an operational evaluation showing whether combined controls can identify synthetic responses while preserving legitimate human participation.

A completed questionnaire is an observed transaction. A genuine human response remains a claim that the research system must substantiate.

Cognaptus: Automate the Present, Incubate the Future.