Wrong Code, Right Data Budget

TL;DR for operators

If verified circuit-training data are scarce, exhaustive validation of every generated design may not be the best use of the next unit of data-engineering budget. In one experiment, a curated 22-circuit synthetic corpus reached 48.49% F1-Micro and 42.82% F1-Macro with a frozen circuit encoder, outperforming the original 22 verified circuits on both metrics and a 110-circuit raw generated corpus on F1-Micro.

The operational lesson is about selection, not permission to train on arbitrary broken hardware. Some generated designs remain structurally informative after synthesis even when their behavior is imperfect. The paper’s pipeline therefore treats synthesizability, structural alignment, and implementation diversity as separate quality controls.

For EDA and hardware-analysis teams, this suggests a different allocation problem: determine how much verification is needed for the downstream task, then spend the remaining budget on generation, structural screening, and diversity. The boundary is substantial. Loose filtering performs inconsistently, globally coupled control errors can erase the relevant signal, and the experiments cover a limited set of generators, models, circuits, and synthesis settings.

The expensive decision is what must be verified

A team building a circuit representation model faces a familiar data problem: realistic gate-level designs are scarce, proprietary, and expensive to annotate. Generating more hardware code is comparatively easy. Establishing that every generated design behaves exactly as intended is not.

That creates a budget decision before it creates a modeling decision. Should the next unit of effort go toward making every generated example functionally correct, or toward identifying which imperfect examples still contain the structural information the model needs?

Cai, Li, Han, and Wang test the second possibility in Wrong Code, Right Structure: Learning Netlist Representations from Imperfect LLM-Generated RTL.¹ Their target is hardware code that specifies how a digital design should behave—register-transfer level, or RTL—and the gate-level netlists produced after synthesis.

The key distinction is that behavioral correctness and training value need not fail together.

A smaller curated corpus beats five times more raw generation

The clearest evidence comes from a comparison using DeepGate4 as a frozen circuit encoder. This experiment is best read as a robustness test across a different representation setting, not as a separate thesis.

The original training set contained 22 circuits and 58,036 nodes. Fivefold raw generation expanded that to 110 circuits and 496,448 nodes. Yet the raw corpus scored only 40.47% F1-Micro, below the original corpus at 45.99%.

Filtering the generated candidates back down to 22 circuits changed the result:

The filtered set improves F1-Micro by 2.50 percentage points and F1-Macro by 7.51 points over the original corpus while using far less training data than the raw synthetic set.

That result changes the scaling variable. Generation volume by itself is not the asset. The asset is generated material that remains aligned with the structural signal required by the downstream task.

The filtering process first removes unsuitable synthesized candidates and then compares learned gate-level organization against a reference design. A pretrained encoder summarizes each netlist, and candidates are screened according to cosine similarity between those structural descriptors. The paper is therefore not treating source-code plausibility as a proxy for quality. It evaluates what survives synthesis.

Imperfect does not mean structurally arbitrary

The broader operator-level experiment supports the same distinction.

At a matched 22-circuit training budget, the verified baseline reaches 97.10% F1-Micro and 90.15% F1-Macro. Across five independently generated datasets, the imperfect synthetic corpora average 96.87% and 89.92%, respectively. The best generated batch reaches 98.23% F1-Micro and 93.79% F1-Macro.

The best run should not be mistaken for an expected outcome; the five-run average is the more informative indicator of reliability. What the repeated experiment establishes is that exhaustive functional validation was not required for the generated corpus to remain competitive with verified training data under this benchmark.

The mechanism proposed by the authors is structural preservation. A semantic mistake can produce the wrong behavior while leaving portions of the synthesized topology, datapath organization, or dependency structure intact. If those surviving patterns are what the representation learner needs for functional identification, the example can still carry supervision.

The paper’s failure analysis also marks where that reasoning stops working. Local errors in memory or data paths may leave class-discriminative organization largely intact. Errors that propagate through control states, fanout, and control-data dependencies can reorganize much more of the circuit. In those cases, “wrong code” also becomes wrong structure.

Diversity helps only after reliability is controlled

Structural screening could create another problem: retaining only candidates that resemble the reference may narrow the training distribution.

The paper addresses this with what it calls architecture voting. After generated designs are synthesizable, an LLM ranks candidates according to differences in operator architecture, datapath organization, and control structure. The purpose is to retain structurally distinct implementations rather than additional near-duplicates.

A cross-architecture test isolates this component. Raw generated data score 92.47% F1-Micro and 91.72% F1-Macro. Architecture voting raises those results to 94.45% and 93.79%.

That is a 1.98-point Micro improvement and a 2.07-point Macro improvement. The experiment functions as an ablation of the voting branch: it shows that deliberate implementation diversity contributes beyond merely having synthesizable generated examples.

The practical operating problem is therefore two-dimensional. Screening must remove structurally unreliable candidates, while selection must still preserve enough architectural variation to improve transfer.

The threshold study shows how narrow the quality gate can become

The zero-shot IP-level experiment pushes the pipeline from arithmetic operators to larger designs. Training specifications are derived from PicoSoC, while evaluation is performed on the held-out NEORV32 system without target-design supervision.

Here the filtering-threshold sweep is a sensitivity test, and it matters operationally because the outcome is not monotonic at loose thresholds.

Random selection reaches 56.90% F1-Micro and 46.38% F1-Macro. Similarity filtering at a threshold of 0.50 raises Micro to 66.50% but lowers Macro to 42.80%. At 0.70, both metrics deteriorate relative to the stronger settings.

High selectivity changes the picture. At $\tau=0.95$, performance reaches 84.46% F1-Micro and 54.96% F1-Macro—27.56 and 8.58 percentage points above random selection.

The paper also audits whether the transfer could simply reflect reproduction of the held-out design. Generated variants show mean RTL MinHash similarity of 0.1375 to the PicoSoC source but only 0.0002 to NEORV32. Their synthesized-netlist descriptors are also closer to the source than the target. This is useful counterevidence to direct target reproduction, although it cannot rule out every form of pretraining exposure to related circuits.

Cognaptus inference: move part of the budget from validation to triage

The paper directly establishes comparative model performance under its reported benchmarks. It does not calculate engineering ROI.

Our inference is that teams with scarce verified netlists should reconsider treating exhaustive functional verification as the universal admission criterion for representation-learning data.

For an EDA team training models to recognize circuit structure, the affected decision is how to allocate data-engineering effort when verification is expensive. Under conditions similar to those tested here, a pipeline could generate a larger candidate pool, require successful synthesis, screen candidates for reference-aligned gate-level structure, and deliberately retain implementation diversity. Full verification could then be concentrated where the downstream use demands behavioral guarantees.

That recommendation does not extend automatically to verification models, safety-critical hardware signoff, or any task whose label depends directly on exact functional correctness. The value demonstrated here is specifically representation supervision for functional identification.

Where this result should stop

The study tests two RTL-generating LLMs, two representation-learning backbones or adaptation settings, one operator benchmark, and one PicoSoC-to-NEORV32 transfer setting. Synthesis uses Synopsys Design Compiler and a TSMC 90 nm standard-cell library. Robustness across other toolchains, technologies, circuit families, and generators remains unresolved.

Selection itself is also sensitive. The IP experiment shows that loose structural thresholds can admit supervision that does not transfer reliably. More fundamentally, the failure analysis indicates that no similarity score can make globally disruptive semantic errors harmless.

The defensible operating rule is therefore narrower than “incorrect synthetic hardware is good enough.” Imperfect generated hardware can be economical training material when synthesis preserves the structural information the downstream model needs, and when the pipeline can identify those cases reliably.

That turns synthetic-data scaling in circuit learning into a curation problem first and a generation problem second.

Cognaptus: Automate the Present, Incubate the Future.