Data Engineering

Enterprise AI usually begins with a deceptively simple request: ask the system a business question and get an answer. Then reality enters, politely carrying a knife. The relevant data is not in one table. The schema is incomplete. The user’s intent depends on personal preference. A term such as “Bay Area” needs external knowledge. A PDF, a web page, an image, and a database record all matter. Someone wants the answer explained, filtered, joined, visualized, and revised after a follow-up question. The demo looked like a chatbot; the production requirement looks suspiciously like distributed systems engineering. ...

Screen. That is where many ambitious AI agents quietly embarrass themselves. Not in a grand philosophical test of intelligence. Not in a graduate-level theorem. Just on a screen: a small button, a chart label, a checkout field, a misread table cell, a tiny icon in a crowded interface. The model can explain strategy, summarize policy, and generate six polite versions of an apology email, but then it clicks the wrong thing because it did not really see the thing. ...

Facts are rude. They rarely arrive in the direction your software needs them. A customer database may know that Alice reports to Bob, while the compliance officer asks, “Who reports to Bob?” A product catalog may store that SKU-17 belongs to Category X, while the chatbot receives, “Show me all products in Category X.” A medical knowledge base may encode one directional relation, while the user asks for the inverse. Humans treat these as the same fact seen from opposite ends. Language models, being very expensive autocomplete machines with a talent for plausible theater, do not always share our confidence. ...

A delayed train is easy to complain about and surprisingly hard to explain. The passenger sees one number: five minutes late, twelve minutes late, cancelled, chaos. The operator sees a messier object. Was the train already late when it entered the station? Did the station itself add delay? Was the delay caused by snow, low visibility, wind, passenger boarding, a single-track bottleneck, equipment failure, or simply the accumulated sins of every previous station on the route? ...

Every enterprise AI conversation eventually reaches the same awkward sentence: “Yes, the agent can write code, but absolutely do not let it touch production.” This is not because executives have suddenly become philosophers of machine autonomy. It is because production data is where optimism goes to be audited. A clever agent that drafts SQL, patches a pipeline, or debugs a transformation is useful right up to the moment it drops a table, joins incompatible versions of data, installs a charmingly malicious package, or writes hallucinated output into a dataset used by finance, compliance, or customer operations. At that point, it is no longer “agentic productivity”. It is an incident report with better syntax. ...

Workshop logs are not glamorous. They are where predictive-maintenance dreams go to meet misspelled component names, missing codes, wrong vehicle identifiers, and dates that imply a truck was both under repair and happily accumulating kilometres. Industrial AI, as ever, is less a matter of elegant algorithms than of persuading messy operational records to stop lying. ...

TL;DR Deep Research agents are good at planning over messy data. They are less good at finishing the plan without taking convenient shortcuts, which is awkward if the job involves recall, auditability, or a CFO who dislikes “probably”. Semantic-operator systems have the opposite problem: they can process unstructured records methodically, but their iterator-style execution can be expensive, slow, and clumsy when the answer requires reasoning across files. ...

TL;DR for operators Most enterprise data work is not blocked by a lack of models. It is blocked by orchestration. A company may already have Spark, Pandas, SQL engines, notebooks, dashboards, semantic layers, data lakes, vector stores, ETL jobs, monitoring tools, and a growing pile of LLM wrappers. The awkward part is deciding which tool should act, in what order, on which data, under which assumptions, and how to recover when the first plan fails. This is the gap the Data Agent paper tries to formalise.1 ...