Code Translation Is Not a Round Trip
TL;DR for operators An analytical workflow can be translated into code that executes successfully while silently changing its results. On ORCA-Main, Claude-Opus-4.6 achieves 75.07% success translating NumPy to Pandas but only 32.93% in the reverse direction—a 42.14-percentage-point gap. The ORCA benchmark1 evaluates this problem across 1,800 data science translation tasks. The asymmetry challenges the assumption that migration is primarily a matter of replacing equivalent API calls. Translation difficulty also depends on how source-library operations can be reconstructed in the target library. Complete workflows introduce another risk: individually correct transformations can still produce incompatible schemas, feature ordering, or preprocessing states. The leading model achieves 56.92% on the full ORCA-Main benchmark and 33.67% on ORCA-Project under a separate evaluation configuration; these scores are not a controlled comparison of project difficulty. ...