Agents in Lab Coats: When LLMs Try to Become Data Scientists

Spreadsheet first. Not the model. Not the agent. Not the impressive diagram with seven tiny boxes labeled “planner,” “executor,” “critic,” “memory,” “tool user,” “reflection,” and, inevitably, “orchestrator.”

In most companies, data science automation begins with something less glamorous: a messy spreadsheet, a half-documented database table, a recurring report, a manager asking why last month’s number changed, and one unlucky analyst trying to remember whether “customer_id” means account, user, buyer, household, or whatever the CRM vendor believed in 2019.

This is the practical entry point for the survey Large Language Model-based Data Science Agent: A Survey.¹ The paper is not presenting a new agent that beats every benchmark. It is doing something less flashy and more useful: organizing the rapidly expanding literature on LLM-based data science agents from two perspectives at once.

The first perspective is the agent-design view: how agents are structured, how they execute tasks, how they use external knowledge, and how they reflect on errors. The second is the data-science workflow view: preprocessing, statistical computation, feature engineering, model training, evaluation, and visualization.

That dual view matters because “AI data scientist” is a dangerously elastic phrase. It can mean a chatbot that writes a Pandas snippet. It can mean an AutoML wrapper with some natural-language dressing. It can mean a multi-agent system that plans, codes, tests, reviews, retrieves documentation, trains models, evaluates outputs, and produces visual reports. These are not the same product. They should not have the same budget, risk rating, or procurement checklist.

The useful question is therefore not: “Can LLM agents do data science?”

The better question is: “Which agent structure fits which data-science workload, and where does each structure fail?”

That is where this survey becomes more than a literature map. Read commercially, it is a quiet warning against the lazy assumption that more agents automatically means better automation. Sometimes more agents create verification. Sometimes they create bureaucracy with API calls. A familiar corporate innovation, really.

The survey’s real contribution is a comparison framework, not a leaderboard

The paper’s main contribution is not one number. It is a way to compare systems.

It organizes LLM-based data science agents across four agent-design dimensions:

The second half of the paper maps these agent-design choices onto the data science loop: data preprocessing, statistical computation, feature engineering, model training, evaluation, and visualization. That mapping is the key move. It prevents the reader from treating “agent architecture” as an abstract software-design preference.

A single-agent tool may be enough for quick exploratory analysis. It may be inappropriate for a regulated reporting pipeline. A dynamic multi-agent system may be useful for open-ended research exploration. It may be exactly the wrong thing for a monthly finance close process where reproducibility matters more than clever improvisation.

The paper’s tables and benchmark review should be read as synthesis evidence, not experimental evidence. Table 3, for example, compares single-agent, two-agent, multi-agent, and dynamic-agent structures across reliability, scalability, coordination cost, predictability, stability, and industrial applicability. That is a framework for architecture selection. It is not a controlled performance test. Table 5 summarizes benchmark suites for data science agents. That is a landscape map. It is not a claim that existing benchmarks fully measure real-world data science competence.

This distinction matters. A survey can discipline product thinking, but it cannot prove vendor ROI. Procurement teams still need pilots, logs, evaluation sets, and failure analysis. The survey tells them where to look.

Single agents are cheap, fast, and fragile in exactly the expected places

The simplest design is the single-agent system. One LLM-based agent receives the task, reasons through it, calls tools, writes code, observes results, and iterates. Many such systems follow the familiar ReAct-style loop: think, act, observe, repeat.

The appeal is obvious. A single-agent system has minimal coordination overhead. It is easier to deploy, cheaper to run, faster to debug, and more suitable for low-risk tasks. If the job is to generate a chart from a clean CSV, explain a small regression output, or write a quick SQL query for an analyst who will inspect the result anyway, a single agent may be good enough.

The problem is equally obvious. The paper notes that raw LLMs remain vulnerable to hallucination, brittle code generation, and long-horizon context failure. In data science, those weaknesses are not cosmetic. They touch the core of the work.

A hallucinated column name is not just an awkward sentence. It is failed code, or worse, silently wrong code. A brittle script is not just an engineering inconvenience. It can corrupt a preprocessing step and contaminate downstream analysis. Context drift is not merely forgetfulness. It means the agent may forget assumptions made earlier in the workflow: how missing values were handled, whether a column was filtered, which model version was evaluated, or why one subset was excluded.

The business interpretation is straightforward:

Single agents are not useless. They are useful precisely when failure is cheap, visible, and reversible. That is not an insult. Most good automation begins in that zone.

The mistake is promoting the same architecture into an end-to-end data-science employee. One agent doing everything is not a data scientist. It is a confident intern with a command line.

Two-agent systems add verification, but only where the second role can actually see the failure

The next step is role separation. The survey highlights two common two-agent patterns: planner-executor and coder-reviewer.

The planner-executor structure separates strategy from implementation. The planner decomposes the task, decides what should happen next, and passes instructions to the executor. The executor interacts with tools, data, and code environments.

The coder-reviewer structure separates generation from criticism. One agent writes code or produces an analytical output. Another reviews it, comments on it, and asks for revision.

This sounds reassuring because it resembles human workflow. Analysts make plans. Engineers implement. Reviewers inspect. Managers nod solemnly in meetings. Civilization continues.

But the paper’s synthesis makes clear that the reliability gain is conditional. A planner can hallucinate a bad plan. A reviewer can miss a subtle statistical error. A code reviewer can catch syntax errors while ignoring leakage in a model evaluation split. A reviewer can also become ceremonial: another LLM politely approving the first LLM’s mistake, now with more tokens.

The real value of two-agent design is not “two is better than one.” It is targeted separation.

Planner-executor helps when the main risk is task decomposition. Coder-reviewer helps when the main risk is implementation error. Neither automatically solves data validity, ambiguous user intent, or weak evaluation design.

For businesses, this is the middle tier of agentic data science automation. It fits predictable workflows where basic verification is valuable but full multi-agent orchestration would be excessive.

Examples include recurring analysis notebooks, controlled data-cleaning tasks, internal reporting scripts, and analyst-assist tools where a human still reviews final outputs. Two agents are especially attractive when the workflow has a repeated shape: load data, clean it, compute metrics, generate chart, check output.

The key is to define what the reviewer is allowed to inspect. If it only sees generated text, it may catch rhetorical nonsense. If it can run code, inspect errors, check data schemas, and compare outputs against expected ranges, it becomes more useful. If it can test statistical assumptions, detect missing-value patterns, and question evaluation design, then we can begin to speak seriously.

Until then, “reviewer agent” may simply mean “a second agent with a slightly more judgmental prompt.” The enterprise software market has sold worse things, but we do not need to applaud.

Multi-agent systems trade simplicity for modular control

Multi-agent systems introduce specialized roles. The survey discusses several families: software-engineering-style teams, minimum-function agents, and client-server structures.

The software-engineering-style design borrows from human development teams. Agents may act as product manager, requirements analyst, architect, developer, senior developer, QA engineer, or tester. The minimum-function design narrows each agent to a small task such as code search, fault localization, patch generation, validation, or exception handling. The client-server design places a central controller over specialized client agents.

This is where reliability can improve. Narrow roles reduce context burden. Specialized agents can check each other’s outputs. The system can decompose a complex workflow into smaller stages. In data science, that matters because the workflow itself is staged: preprocessing affects feature engineering, feature engineering affects model training, model training affects evaluation, and evaluation affects business interpretation.

A well-designed multi-agent system can mirror this structure.

This is the most business-relevant design for production analytics. It provides modularity, stage-level control, and clearer audit trails. If a monthly churn model fails, the team can investigate whether the issue came from data ingestion, feature construction, training, evaluation, or reporting. That is much better than asking a single agent, after the fact, why it did something three steps ago. It may answer beautifully. It may even be wrong beautifully.

But multi-agent systems introduce their own cost. Communication overhead rises. Latency increases. Miscommunication becomes a system-level risk. The workflow becomes harder to design and maintain. A controller agent can route tasks incorrectly. Two specialized agents may produce incompatible assumptions. A local correction can create a global inconsistency.

This is why the survey’s comparison is useful: multi-agent systems score better on reliability and scalability, but weaker on coordination cost. That trade-off is not a footnote. It is the product decision.

For business use, multi-agent architecture is justified when the workflow is valuable enough, repeated enough, and risky enough to deserve modular controls. A CFO dashboard, credit-risk model, drug-discovery pipeline, or production forecasting process may justify the complexity. A one-off chart for a meeting probably does not.

Dynamic agents are excellent for exploration and awkward for production stability

Dynamic agent systems go further. Instead of relying only on predefined roles, they create, modify, or expand agents during runtime. The paper describes two broad patterns: hierarchical agent generation and iterative agent generation through feedback.

In hierarchical generation, a parent agent decomposes the task and creates child agents for subtasks. In iterative generation, the system adjusts its structure based on feedback, adding or modifying agents as the task evolves.

This is intellectually attractive because data science projects often change as they unfold. A dataset reveals unexpected missingness. A first model performs badly. A visualization exposes a strange subgroup. A user clarifies that the real question was not “forecast revenue” but “explain why revenue forecast accuracy collapsed after a pricing change.” Naturally, a rigid pipeline may struggle.

Dynamic agents are therefore appealing for exploratory analytics, research prototyping, and ambiguous problem spaces. They can adapt to new information, branch workflows, and create specialized sub-agents as needed.

The cost is predictability. Dynamically spawned agents amplify uncertainty. They make reproducibility harder. They increase compute and coordination overhead. They can also make governance uncomfortable because the system’s structure is not fully known in advance.

For production settings, this matters. If a model used in a regulated process produces a bad decision, “the agent dynamically created another agent” is not a satisfying audit explanation. It is the AI equivalent of “someone from another department handled it.”

Dynamic agents are best treated as research accelerators, not default production architecture. They belong in sandboxes, innovation labs, exploratory analytics, and rapid pipeline prototyping. Once a workflow stabilizes, the business should consider freezing useful parts into more predictable structures: static stages, explicit tests, versioned prompts, reproducible notebooks, controlled tools, and audit logs.

Dynamic discovery first. Structured operation later. This is less glamorous than autonomous everything, but reality often has terrible branding.

Execution structure decides whether the system is reproducible or adaptive

The survey’s second major design axis is execution structure. A system may follow static execution, dynamic execution, or hybrid execution.

Static execution means the workflow is predefined. The agent follows a sequence: inspect data, clean data, engineer features, train model, evaluate model, produce report. Static workflows are easier to evaluate and reproduce. They are useful where consistency matters.

Dynamic execution means the agent adjusts its next steps based on observations. The paper discusses just-in-time planning, where an agent observes the result of a step and decides what to do next, and plan-then-execute patterns, where a plan is created and then executed or refined.

For business purposes, static and dynamic execution answer different needs.

The interesting business design is often hybrid. A company may define a fixed reporting pipeline but allow dynamic exception handling when the agent detects schema drift, missing values, or unusual metric changes. The agent is not free to redesign the entire workflow whenever it feels inspired. It is allowed to branch only under defined conditions.

That is a more realistic model of autonomy: not “do whatever seems best,” but “operate within bounded discretion.”

External knowledge is not decoration; it is how agents stop pretending the prompt contains the world

The survey groups external knowledge into external databases, retrieval-based methods, API calls, search-engine integration, and hybrid approaches.

This is critical for data science because internal model knowledge is rarely enough. Data science agents need current data, domain documentation, database schemas, code repositories, statistical libraries, model APIs, business definitions, and sometimes regulatory or scientific references.

A model may know what a t-test is. It does not automatically know how the company defines “active customer,” where the latest table lives, whether the schema changed last quarter, or which KPI definition finance uses in board reporting.

External knowledge changes the agent from a fluent guesser into a tool-using system. But the type of knowledge source should match the task.

For enterprises, the key question is not whether an agent has RAG. That has become a checkbox. The question is whether the retrieval layer is connected to the right knowledge, whether the agent knows when to use it, and whether retrieved information is traceable.

An agent that retrieves irrelevant documentation is not grounded. It is distracted with citations.

Reflection is useful only when it can inspect the right object

Reflection is one of the most abused words in agent design. In weak versions, it means the model is asked to “think again” after producing an answer. That can help with simple mistakes. It can also produce a second confident answer with better grammar.

The survey treats reflection more systematically. Reflection mechanisms include agent feedback, code error handling, unit testing, model-metrics feedback, history windows, and human feedback. It then classifies reflection by driver, level, and adaptability.

That classification is valuable because not all reflection catches the same failures.

The paper’s future-research section makes a particularly important point: current reflection often operates at the step level. It checks the latest code snippet or response. But data science errors often propagate across stages. A bad merge upstream may produce a plausible model downstream. A missing-value decision may distort a later statistical estimate. A leakage problem may produce excellent test metrics, which is precisely the problem.

This is why pipeline-level reflection matters. A serious data science agent should not only ask, “Did this code run?” It should ask:

• Did the data schema match the assumed schema?
• Did preprocessing change the population being analyzed?
• Did feature engineering leak target information?
• Did evaluation match the business objective?
• Did the final chart represent the model output honestly?
• Did an upstream uncertainty become a downstream false certainty?

That is the difference between code repair and analytical reliability.

The benchmark section is a warning label for easy demos

The paper reviews a wide range of benchmark suites for data science agents, covering general-purpose data analysis, machine learning, visualization, geospatial analysis, biomedical tasks, company-style work, and other specialized settings.

The benchmark table is useful because it shows how fragmented the evaluation landscape is. Some benchmarks test broad analytical skills. Some focus on visualization. Some test domain-specific reasoning. Some test end-to-end corporate tasks. Task counts vary widely, from small research-style suites to thousands of generated or collected tasks.

But the paper also points out structural limitations. Many benchmarks evaluate short, isolated tasks. Some have rigid formats that allow models to exploit templates. Many inputs are cleaner and more explicit than real business data. These conditions can make an agent look more competent than it would be inside a live data workflow.

This is the familiar demo problem. A model performs well on a clean benchmark task, then enters a company environment where tables are inconsistently named, definitions have changed, credentials fail, APIs timeout, stakeholders disagree, and the “ground truth” is a spreadsheet maintained by someone named Brian who left in 2021.

Benchmarks are necessary. They are not sufficient.

A practical evaluation stack for business use should include at least four layers:

The survey does not give companies a ready-made evaluation framework. It explains why one benchmark number should not be trusted as a full measure of data-science competence. That is already a service.

The business decision map: match architecture to risk, not fashion

The survey’s most practical implication is architectural matching. Different data-science tasks require different agent structures.

This is the business relevance pathway: the paper should be used as a design and procurement checklist. A buyer should not ask vendors only whether they “use agents.” That question is too vague to be useful.

Better questions include:

• How many roles exist, and what does each role inspect?
• Is execution static, dynamic, or hybrid?
• What external knowledge sources are connected?
• Can the system run code and observe execution results?
• Does reflection inspect text, code, data, metrics, or the whole pipeline?
• How are intermediate assumptions logged?
• Can the workflow be reproduced?
• What happens when the agent is uncertain?
• What benchmark results exist, and what do those benchmarks fail to test?
• Where does human review enter the process?

These are not glamorous questions. They are the questions that prevent a company from buying a “data science agent” and receiving a chatbot with a Python habit.

What the paper directly shows, what Cognaptus infers, and what remains uncertain

It is worth separating the layers of interpretation.

The paper directly shows that the literature on LLM-based data science agents can be organized through a dual perspective: agent design and data-science workflow. It directly synthesizes common role structures, execution patterns, knowledge-integration methods, reflection mechanisms, workflow stages, benchmark suites, and future research needs.

Cognaptus infers that this framework can support business architecture decisions. The survey’s comparison of single-agent, two-agent, multi-agent, and dynamic-agent systems maps naturally to procurement and deployment choices. Low-risk tasks can use simpler structures. Production workflows require stronger modularity and verification. Exploratory work may benefit from dynamic agents, but production use needs reproducibility and control.

What remains uncertain is performance magnitude. The survey does not prove that a particular architecture produces a specific ROI improvement. It does not provide a universal benchmark result showing that multi-agent systems always outperform simpler systems. It does not solve the operational challenge of integrating agents with messy enterprise data, permissions, governance processes, and human accountability.

That uncertainty is not a weakness of the paper. It is the boundary of what a survey can responsibly claim.

The next frontier is not more agents; it is better failure visibility

The paper’s future-research directions are unusually practical: data-centric diagnostics, uncertainty-aware workflow planning, and pipeline-level reflection.

Data-centric diagnostics address failures that originate in the data itself: irregular schemas, semantic mismatches, missing-value structures, distribution shifts, and inappropriate assumptions. This is essential because many data science failures are not reasoning failures in the narrow sense. The agent may reason correctly over the wrong representation of the data.

Uncertainty-aware workflow planning addresses the problem of false confidence. If an agent is unsure what a column means or whether a user’s request is underspecified, it should not push forward as if nothing happened. It should ask for clarification, branch the workflow, retrieve more context, or enter a stronger verification mode.

Pipeline-level reflection addresses the gap between local fixes and global coherence. A system that corrects a code snippet but never revisits the earlier data decision is not truly reflective. It is tidying the last room while the basement floods.

These directions point to a more mature view of data science agents. The goal is not to create a swarm of agents performing theater. The goal is to make analytical workflows more observable, correctable, and aligned with business decisions.

Conclusion: the lab coat does not make the scientist

LLM-based data science agents are moving from toy demos toward serious workflow automation. The survey gives us a useful vocabulary for judging that movement: role design, execution structure, external knowledge, reflection, workflow coverage, benchmark scope, and future reliability mechanisms.

The central lesson is simple: agent architecture is a risk-control design, not a decoration.

Single agents are useful when speed matters and failure is cheap. Two-agent systems add modest verification when the roles are well-defined. Multi-agent systems become attractive when workflows are valuable, repeated, and auditable. Dynamic agents are powerful for exploration but awkward for stable production unless their outputs are later disciplined into reproducible pipelines.

More agents do not automatically create more intelligence. Sometimes they create more places for mistakes to hide.

A good data science agent should not merely generate code, call tools, and narrate results. It should understand where uncertainty enters the workflow, where data defects can poison the analysis, where local fixes are insufficient, and where human judgment is still required.

That is the difference between an agent wearing a lab coat and an agent that can actually work in the lab.

Cognaptus: Automate the Present, Incubate the Future.