Code Translation Is Not a Round Trip

TL;DR for operators

An analytical workflow can be translated into code that executes successfully while silently changing its results. On ORCA-Main, Claude-Opus-4.6 achieves 75.07% success translating NumPy to Pandas but only 32.93% in the reverse direction—a 42.14-percentage-point gap. The ORCA benchmark¹ evaluates this problem across 1,800 data science translation tasks.

The asymmetry challenges the assumption that migration is primarily a matter of replacing equivalent API calls. Translation difficulty also depends on how source-library operations can be reconstructed in the target library. Complete workflows introduce another risk: individually correct transformations can still produce incompatible schemas, feature ordering, or preprocessing states. The leading model achieves 56.92% on the full ORCA-Main benchmark and 33.67% on ORCA-Project under a separate evaluation configuration; these scores are not a controlled comparison of project difficulty.

For data platform owners, automated migration should be accepted only after validating required behavior and interfaces between pipeline stages. Explicit descriptions of source-code intent and execution-feedback agents can improve translation success, but neither guarantees correctness. More elaborate agent workflows also incur substantially higher inference costs.

A Successful Execution Is Not a Successful Migration

Consider a data team replacing an existing Pandas workflow with NumPy code. The translated implementation executes without errors and produces an array of the expected dimensions. Yet the ordering of features has changed, affecting every downstream prediction.

The implementation is syntactically valid and operationally executable, but it no longer preserves the original workflow’s required behavior.

This distinction becomes consequential when organizations migrate SQL queries, modernize analytical libraries, or replace machine learning frameworks. Translation accuracy depends on whether the target implementation preserves data values, aggregation logic, types, ordering, numerical behavior, and model state.

ORCA addresses this problem through functional equivalence: two implementations must satisfy the same specified behavioral requirements, rather than merely resemble one another or execute successfully.

The benchmark contains two complementary evaluations:

ORCA-Main includes 600 querying tasks, 600 manipulation tasks, and 400 deep learning tasks. ORCA-Project examines complete workflows with dependencies between components.

Translations are checked against executable tests and manually annotated reference implementations. Success requires passing all applicable tests, with reported primary results averaged across three evaluation runs.

Among 16 models evaluated on the full ORCA-Main benchmark, Claude-Opus-4.6 leads at 56.92%, compared with 56.42% for Gemini-3.1-Pro. Claude-Opus-4.6 achieves 33.67% in the separate eight-model ORCA-Project evaluation.

These figures reveal substantial reliability gaps. They should not, however, be treated as a controlled estimate of the additional difficulty caused by project length: the two benchmark settings use different tasks and default inference configurations.

Reversing the Libraries Changes the Difficulty

One particularly revealing result comes from translating between NumPy and Pandas.

On ORCA-Main, Claude-Opus-4.6 achieves:

Source: Appendix H, Table 12.

The model, libraries, and broad task domain remain the same. Reversing the direction nevertheless produces a substantial performance difference.

ORCA investigates this asymmetry through operation correspondence: whether one source-library operation can be reproduced by a single target-library operation or requires several operations to reconstruct equivalent behavior.

In the operation-level analysis, 57.95% of Pandas-to-NumPy source-operation occurrences require indirect or multi-step reconstruction. The corresponding proportion for NumPy-to-Pandas is 34.20%.

Pandas provides comparatively high-level operations for labeled data, indexing, and tabular transformations. Reconstructing such behavior using lower-level array operations can require additional implementation decisions. Conversely, explicit numerical operations may translate more directly into higher-level constructs.

The broader evaluation also finds that Pandas-to-PostgreSQL and TensorFlow-to-PyTorch outperform their reverse directions across the evaluated models.

This supports an association between library abstractions and translation difficulty, but operation correspondence alone does not explain every result. Framework execution structures and other semantic differences may also matter.

For migration planning, source and target libraries cannot be treated as an unordered pair. A successful evaluation of one direction supplies limited evidence about reliability in the reverse direction.

Correct Components Can Produce an Incorrect Project

Project-level translation introduces another requirement: components must continue to exchange compatible data.

An individual transformation may compute the expected values while returning a different schema. A model may train successfully but receive features in a different order during inference. Preprocessing may be correct locally yet inconsistent between training and prediction.

ORCA-Project examines these dependencies using complete analytical workflows.

The authors measure input-data complexity as the number of input sources multiplied by the number of flattened data fields. Across four analyzed models, average success decreases as this proxy increases:

Source: Appendix G, Table 10. Each average covers four selected models.

A supplementary within-category analysis finds a similar decline among classification projects, from 36.25% in the low-complexity group to 15.91% in the high-complexity group. This checks whether the broader association persists within a narrower task category; it does not establish that data complexity independently causes the failures.

The failure taxonomy provides a more direct diagnosis.

Among analyzed failed ORCA-Project translations from four representative models, 38% are classified as cross-stage interface failures and another 38% as local data-semantic failures. Output or project-contract violations account for 13%, while modeling, implementation, or execution errors account for 11%.

Source: Appendix I.4.3, Table 23. These are mutually exclusive primary classifications of analyzed failures, not percentages of all benchmark tasks.

Cross-stage failures include schema differences, incompatible types, feature ordering, and training–inference misalignment. Local failures involve incorrect filtering, aggregation, parsing, missing-value treatment, or feature transformations.

The distinction changes how translated projects should be tested. Unit tests establish local behavior; integration tests must separately establish compatibility between producers and consumers. Passing one does not imply passing the other.

Explicit Intent Helps More Reliably Than Simply Adding Reasoning

ORCA also evaluates whether translation improves when the model first explains the source code’s intended functionality.

Its two-stage Intent-Augmented method generates a natural-language description of source behavior before producing the target implementation. For complete projects, intent is represented at both module and project levels.

Across five evaluated models, this approach improves average success over baseline by:

• 4.80 percentage points on ORCA-Main.
• 5.33 percentage points on ORCA-Project.

Source: Section 5.5, Table 3.

The gains are meaningful but heterogeneous. On ORCA-Main, Qwen2.5-Coder-32B increases from 30.88% to 38.71%, while Claude-Opus-4.6 increases from 56.92% to 60.92%.

The researchers also test Chain-of-Thought intent generation. Its performance is model-dependent: Claude-Opus-4.6 reaches 62.71% with CoT-Intent, exceeding its concise-intent result, whereas Qwen2.5-Coder-32B declines to 29.50%, below its 30.88% baseline.

The distinction is between making functional requirements explicit and generating more reasoning text. A longer intermediate description can introduce inaccurate assumptions that subsequently influence implementation.

For developers building migration assistants, Cognaptus infers that structured intent extraction deserves evaluation before investing in model retraining. This is a workflow-design opportunity supported by observed prompting gains, not evidence that any particular intent representation guarantees correctness.

Execution Feedback Improves Accuracy at a Substantial Cost

A coding agent can inspect translated files, execute candidate implementations, examine failures, and revise code. ORCA evaluates whether these capabilities improve translation outcomes.

In a supplementary comparison using 300 representative ORCA-Main tasks, a ReAct-style execution-feedback agent raises Claude-Opus-4.6’s success rate from 43.33% under direct prompting to 49.00%. The agent is permitted up to 20 interaction turns.

A separate Claude Code evaluation produces larger improvements, with considerably higher inference costs:

Source: Appendix I.3.4, Table 19. Costs are in USD for the evaluated subsets.

The Claude Code configuration costs approximately 30 times the baseline for the Main subset and 3.5 times for the Project subset.

These are supplementary agent-configuration comparisons, not replacements for the primary full-benchmark rankings. A separate model comparison also finds GPT-5.5 with xhigh reasoning outperforming Claude-Opus-4.6 on a matched 300-task subset, emphasizing the importance of task selection and inference settings when interpreting model rankings.

For organizations evaluating autonomous migration tools, the economic question concerns the incremental cost of obtaining additional verified successes. The benchmark provides measured cost and accuracy differences, but production value also depends on human review costs, the consequences of undetected errors, and the availability of representative tests.

Migration Acceptance Should Follow Behavioral Requirements

ORCA suggests a practical division of responsibilities for data platform and MLOps teams.

Before translation, specify required behavior: numerical outputs, null handling, data types, ordering, model-state behavior, and externally visible interfaces. Identify operations likely to require multi-step reconstruction in the target library.

During translation, evaluate whether an explicit functional-intent description improves results for the selected model and source-target direction. Where justified, permit execution feedback and iterative repair.

Before deployment, test both individual transformations and complete producer-consumer workflows. Include boundary inputs, schema variations, preprocessing-state consistency, and train–inference alignment where relevant.

This is a Cognaptus operational interpretation of the benchmark, rather than a production procedure experimentally validated by the authors. Its value lies in assigning different validation requirements to local computation, intermediate interfaces, and complete workflow behavior.

Two boundaries remain important. ORCA covers selected libraries and analytical tasks rather than every enterprise ecosystem; the authors identify technologies such as R and Spark as potential extensions. Moreover, passing a finite executable test suite demonstrates correctness against the tested properties, not unrestricted equivalence across all possible inputs.

The Acceptance Standard Must Be Stronger Than Executable Code

ORCA’s central contribution is a measurable distinction between generating plausible translations and preserving the behavior of analytical systems.

The directional asymmetry shows that the representation of source operations matters. Project-level failures show that correct local behavior is insufficient when interfaces are inconsistent. Intent extraction and coding agents improve selected outcomes, but substantial errors remain.

For organizations modernizing data science infrastructure, the resulting principle is specific: automated translation can generate candidate implementations, while acceptance must depend on independent behavioral and cross-stage validation.

The relevant measure of migration success is the preservation of required analytical behavior under tested conditions, not whether the translated program can run.

Cognaptus: Automate the Present, Incubate the Future.