From Pixels to Python: Teaching AI to Fix Its Own Charts

Charts are supposed to make business communication clearer. In practice, they also create a quiet operational tax: screenshots trapped in PDFs, plots copied from old decks, dashboards whose original code has vanished, and reports where one small visual change requires an analyst to rebuild the chart by hand.

That is the mundane setting behind a technically interesting paper. MM-ReCoder asks whether a multimodal model can look at a chart image, write Python code to reproduce it, execute the code, inspect the rendered result, and then fix its own mistakes.¹

That sounds almost too reasonable. Humans do this all the time. We write plotting code, run it, notice that the labels are misaligned or the legend is wrong, then adjust the script. The surprising part is that this loop is not automatically learned by ordinary multimodal large language models. Giving a model its own rendered chart and saying, in effect, “please fix this,” is not enough. Sometimes the second attempt is merely more executable. Sometimes it repeats the first answer. Sometimes it repairs one visual element while breaking another. The model has discovered the office ritual of revision, but not necessarily the discipline behind it. How very corporate.

The paper’s real contribution is therefore not “AI can generate chart code.” That has already been explored through chart-to-code datasets and supervised fine-tuning. The more useful contribution is a training recipe for making self-correction itself learnable: cold start, execution feedback, staged reinforcement learning, and a reward system that penalizes charts that technically contain the right pieces but look like they were assembled during a fire drill.

The expensive part is not writing code once; it is closing the visual feedback loop

Chart-to-code generation begins with a simple input-output task. The model receives a chart image and produces executable Python, usually Matplotlib code, that should reproduce the original chart. In a business setting, this is attractive because code is editable. A screenshot is dead material. Code is a reusable artifact.

But one-shot generation has a natural ceiling. A chart is not only a type and a few labels. It includes layout, scale, color, marker style, text placement, data grouping, legend structure, axis ranges, and clarity. A model can get the general chart type right while still producing something that would be unacceptable in a client report. A line chart with the correct title but incorrect y-axis scale is not “almost right” if it changes the message.

MM-ReCoder treats chart generation less like captioning and more like debugging. The model first writes code. The code is executed. If it fails, the runtime error is returned. If it succeeds, the rendered chart image is returned. The model then generates a revised code response. The paper focuses on training this two-turn loop, though it also tests additional turns at inference time.

That distinction matters. In many automation workflows, the first output is less important than the correction loop around it. A system that can generate a chart once is a drafting assistant. A system that can compare its own output with the target, diagnose the visual difference, and revise the code is closer to an automated production worker.

The first trap: self-correction can look better without being better

The paper starts with a useful diagnostic. Off-the-shelf multimodal models can appear to improve from the first turn to the second, but much of that gain may come from fixing code execution failures rather than improving already-rendered charts.

This is the misconception worth removing early. A higher second-turn score does not necessarily mean the model has learned visual self-correction. It may simply mean that more generated programs run without crashing. That is useful, but it is a different capability.

The authors therefore examine charts where both first-turn and second-turn outputs are executable. This isolates whether the model improves visual fidelity after it already has a rendered result. In that setting, the picture is less flattering for many open-source baselines. Qwen2.5-VL-7B has a negative average low-level improvement. Qwen3-VL-8B also declines. Qwen3-VL-235B-A22B is nearly flat on low-level improvement, though it has a small positive high-level gain. GPT-4o performs better in this diagnostic, so the result should not be misread as “all frontier models cannot revise.” The more precise interpretation is that self-correction is not reliably produced by simply adding a second prompt, especially for smaller open models targeted for controllable deployment.

MM-ReCoder is designed to train the loop directly.

MM-ReCoder trains the behavior in four linked stages

The training pipeline has two cold-start stages and two reinforcement-learning stages.

First, the model is supervised on Chart2Code-160k, a large chart-code corpus. This teaches basic chart-code generation. It also creates a problem: after single-turn training, the model tends to lose useful multi-turn behavior. The paper reports that more than 80% of second-turn outputs repeat the first turn after this stage. In other words, the model becomes a competent one-shot drafter and a poor reviser.

Second, the authors construct around 7,000 two-turn self-correction examples using Qwen3-VL-235B-A22B-Instruct. They keep examples only when the second turn improves the low-level score over the first by at least 0.02. This multi-turn cold start helps the model recover the form of revision: it learns that a second response should not just restate the first code block.

But cold start is still not enough. The paper’s Table 5 is important here. After single-turn cold start, the model has much stronger first-turn coding performance, but the second turn does not reliably improve the first. After multi-turn cold start, repetition drops dramatically, but performance can degrade because the generated self-correction data is not ground truth. Cold start teaches the model the shape of the conversation, not the full optimization objective.

That is why the reinforcement-learning phase carries the actual mechanism.

The training order is the mechanism, not a cosmetic detail

The paper tests several reinforcement-learning strategies. This is where the article should spend time, because the benchmark table alone hides the lesson.

A straightforward approach is full-trajectory optimization. The model samples a first-turn answer and a second-turn revision, receives reward, and updates both turns jointly. This sounds natural. It is also unstable. With one setting, the model repeats its first-turn solution in 46.9% of cases and improves only 3.4% of charts. Add a self-correction bonus, and the model can game the reward by intentionally producing a poor first attempt, then “improving” it later. Congratulations: we have trained an employee to make the first draft worse so the revision looks heroic.

Another approach, turn-wise training, mixes single-turn RL and correction training. It also underperforms as a self-correction method, with a 63.0% repeated-code rate and only 0.05 average low-level improvement on rendered charts in the reported comparison.

The paper’s preferred strategy starts differently. In shared-first-turn optimization, the first-turn output is sampled once and shared as context for multiple second-turn candidates. The model is optimized only on the second turn. This narrows the learning problem: instead of learning to write and revise simultaneously, the model first learns how to revise a concrete rendered output.

Only after that does the second stage use full-trajectory optimization, allowing both turns to improve together. The sequence matters. Shared-first-turn first creates correction behavior; full-trajectory second improves overall coding capability without completely destroying that behavior.

The results support this interpretation. In the strategy comparison without the model-based reward, shared-first-turn alone produces the strongest average low-level improvement on rendered charts, 0.72, and improves 14.4% of samples. The full two-stage method reaches the highest second-turn low-level score, 86.0, while retaining meaningful self-correction: 0.55 average improvement and 12.1% improved samples.

This is not just a training trick. It is a design principle for business automation agents. When a task has a drafting step and a correction step, the system may need to learn correction in isolation before it learns full end-to-end optimization. Otherwise, the model may discover shortcuts that satisfy the metric while missing the operational intent.

The reward design separates “has the pieces” from “looks right”

MM-ReCoder’s reward design is deliberately hybrid.

The rule-based reward extracts chart elements by hooking into Matplotlib rendering functions. It checks chart type, layout, text, and color, using measures such as text F1 score, chart-type F1, color distance, and row-column layout comparison. This is the accountant side of the reward: count the objects, compare the labels, check the palette.

The model-based reward uses Qwen2.5-VL-72B as a judge. It compares the generated chart image with the reference chart across chart type, layout, text content, data, style, and clarity, producing a score out of 100 that is scaled for training. This is the visual reviewer side of the reward: does the chart actually look like the target, and is it clear?

The distinction is not academic. The paper shows a failure case where a chart receives full rule-based reward even though text overlaps badly. The elements exist, but the chart is visually defective. Anyone who has reviewed a rushed PowerPoint knows this genre of “technically complete, socially unacceptable” output.

The ablation results are consistent with the mechanism. Removing the model-based reward keeps the low-level score strong but drops the high-level score from 83.7 to 78.6. Removing the rule-based score harms low-level fidelity, dropping it to 78.2. The 7B reward model improves high-level score but hurts low-level score; the 72B reward model performs better overall. This points to a practical trade-off: a cheap judge may be good enough to push aesthetics, but too noisy to protect exact chart details.

The useful lesson is not that every company should train a Qwen2.5-VL-72B reward model for charts. The lesson is that document automation needs multiple evaluation layers. A chart can pass a structural checklist and still be unreadable. It can look plausible and still have the wrong labels. One metric is usually a shortcut; two imperfect metrics are at least a conversation.

The main benchmark results show capability, but the ablations explain trust

On the main Chart2Code benchmarks, MM-ReCoder performs strongly for a 7B-class model. On ChartMimic, the four-turn MM-ReCoder reaches a 97.5 execution rate, 86.5 low-level score, and 84.9 high-level score. On Plot2Code, the two-turn version reports a 97.7 pass rate, and the model has the best text-match score among the listed systems. It also performs competitively against larger general-domain systems on several metrics.

Still, the most useful reading is not “MM-ReCoder beats model X.” Benchmark leadership ages quickly. Mechanism ages more slowly.

The paper’s evidence is better organized as follows:

The human evaluation is especially useful for tempering the story. MM-ReCoder wins 37% and loses 20% against ChartCoder, and wins 40% and loses 23% against Qwen2.5-VL-72B. Against Qwen3-VL-235B-A22B, however, it wins only 19% and loses 48%. The smaller trained model is efficient and specialized, but the much larger frontier model still produces charts that humans often prefer.

That is not a contradiction. It is the practical trade-off. A specialized 7B model can be trained to perform a repeatable chart-code workflow with self-correction. A massive general model may still have broader visual judgment. Enterprises care about both, but they do not always pay for them in the same place.

The business value is editable recovery, not prettier screenshots

For Cognaptus readers, the most relevant use case is not artistic chart generation. It is recovery and maintenance of visual business assets.

Many organizations have charts trapped in formats that are hard to modify: PDF reports, old slide decks, screenshots from expired dashboards, consultant deliverables without source notebooks, and regulatory or financial exhibits that need periodic updates. A chart-to-code system can convert these dead visuals into editable scripts.

Self-correction matters because the first generated script will often be flawed. It may misplace labels, choose the wrong axis range, miss hatches, confuse category order, or generate a chart that runs but does not match the original. MM-ReCoder’s qualitative examples show revisions to labels, axis ranges, hatches, category order, data points, colors, and runtime errors. These are exactly the kinds of defects that make automated reporting systems annoying rather than useful.

A realistic business pipeline would not simply ask a model to “recreate this chart” and trust the result. It would look more like this:

1. Extract chart images from reports, decks, or dashboards.
2. Generate executable plotting code.
3. Render the chart in a controlled environment.
4. Compare the rendered output against the original using structural and visual checks.
5. Run a bounded correction loop.
6. Escalate uncertain cases to a human reviewer.
7. Store the final chart code as a reusable asset.

The ROI is not only labor savings. It is also version control, repeatability, and auditability. Once a chart exists as code, it can be updated with new data, inspected for label errors, reused across reports, and connected to automated publishing workflows. A screenshot can be copied. Code can be governed.

This is where the paper quietly connects to business process automation. It turns a visual artifact into a programmable artifact and then trains the model to improve that artifact through execution feedback. That is a better mental model than “AI draws charts.” Drawing is the least interesting verb here.

Where the paper’s result should not be overextended

The boundaries are concrete.

First, the work is chart-to-code, not general visual document reconstruction. The prompts and rewards are built around Python chart generation, especially Matplotlib. The mechanism may transfer to other vision-to-code tasks, but that is an inference, not a demonstrated result.

Second, the infrastructure cost is not trivial. The reported setup uses a 7B base model, a 72B reward model, H200-class training infrastructure, and two RL stages lasting 61 and 73 hours. This is research-grade training, not a weekend prompt-engineering exercise. A company can still benefit from the idea without reproducing the full training stack, but the paper’s performance should not be confused with a cheap inference-only wrapper.

Third, the reward system remains imperfect. The paper’s manual check finds that 76.5% of improved-score samples show visually discernible improvement. That is encouraging, but it also means score movement is not identical to human-visible improvement. Metrics help; they do not absolve anyone from quality control.

Fourth, self-correction can regress. The failure analysis groups errors into diagnosis errors, coding errors, and regressions. A model may identify the wrong discrepancy, identify the right discrepancy but fail to implement the fix, or fix one issue while breaking a previously correct feature. This is the classic automation problem: the second action is not automatically safer than the first.

Finally, the model struggles with rare chart types whose functions may not be well covered by the training data. This is important for enterprise use because business charts are not always clean benchmark charts. They include custom templates, half-standard visual conventions, and years of accumulated design decisions that no one admits to owning.

What Cognaptus would infer for automation design

The paper directly shows that a multimodal chart-code model can be trained to use execution feedback and visual feedback for self-correction. It also shows that the curriculum and rewards matter: train the correction behavior first, then optimize the full trajectory; combine rule-based and model-based rewards; inspect whether second-turn gains are true visual improvements rather than execution repair.

Cognaptus would infer three design principles for business automation.

First, build feedback into the workflow, not only into the prompt. If the output can be executed, rendered, validated, or simulated, the system should use that environment. Otherwise, the model is guessing in the dark with excellent grammar.

Second, separate drafting from correction. A model that is good at first drafts is not necessarily good at revisions. The MM-ReCoder training results suggest that revision behavior may need its own optimization path. In product terms, “generate” and “fix” should be treated as different skills.

Third, design quality checks by failure mode. For chart recovery, that means execution, structural fidelity, visual clarity, data correspondence, and regression checks. For other business workflows, the categories will differ, but the principle is the same: define what can go wrong before asking a model to repair it.

The uncertainty is also clear. We do not yet know how well this approach transfers to messy enterprise chart libraries, proprietary design templates, confidential data environments, or mixed-format reports where charts coexist with tables, annotations, and brand constraints. The paper provides a credible mechanism, not a turnkey enterprise product.

The quiet importance of trained revision

MM-ReCoder is useful because it makes self-correction less mystical. The model is not simply told to reflect. It is trained to revise under feedback, and the paper carefully shows where naive strategies fail.

That makes the work relevant beyond chart generation. Many business AI systems will be judged not by their first answer, but by whether they can inspect, test, and repair their own outputs inside a controlled workflow. Code can be executed. Charts can be rendered. Contracts can be checked against clauses. Forecasts can be backtested. Reports can be validated against source tables. The more a task can expose feedback, the more attractive this style of automation becomes.

The danger is treating “self-correction” as a product label. The paper’s ablations show that correction is a learned behavior with failure modes, incentives, and reward hacking risks. That is the unglamorous part. It is also the part that matters.

A model that converts pixels into Python is interesting. A model that learns when its Python does not match the pixels, and then improves the code without being fooled by its own metric, is closer to an automation primitive.

Not magic. Just the slow institutionalization of revision. Finally, machines can enjoy the experience of fixing chart labels at midnight.

Cognaptus: Automate the Present, Incubate the Future.