More Synthetic Data Wasn’t Better

TL;DR for operators

Catalog teams often need more labeled examples before an attribute extractor works reliably in a new category or marketplace. Producing additional product-like text is not the difficult part. The training record has to change the intended attribute while leaving unrelated product information coherent.

Negri, Martínez Gómez, Balanya, and Rajaram test a controlled generation pipeline built around that requirement.¹ In their downstream experiment, synthetic-only training reached 60.48% attribute-extraction accuracy, almost identical to 60.79% for original-only training. But the best tested configuration was 75% original + 25% synthetic, at 68.82%. Accuracy declined as the synthetic share rose to 50% and 75%.

For operators, this makes synthetic data an allocation and quality-control problem. Teams need to decide what supervision state they are generating, verify that unrelated information remains intact, and measure how much generated data improves a real training mixture. The paper shows that controlled synthetic listings can carry substantial training signal. It does not show that they should replace real catalog data.

The labeling bottleneck is a control problem

A synthetic product can read perfectly well and still be bad supervision.

Suppose a team needs another example for sleeve style. If a generator changes the sleeve description but also alters material, occasion, or other product facts, the listing may remain plausible while encoding several undocumented changes. An extractor trained on that record no longer has a clean relationship between the structured attribute and its textual evidence.

The framework therefore starts from an existing product rather than generating a listing independently from scratch. It selects one category-relevant attribute and modifies the product around that target while constraining unrelated content.

That is the operational meaning of controlled attribute intervention: create a new training case by changing one intended property while retaining enough of the original record to know what was supposed to stay fixed.

The pipeline constructs three different supervision states

The framework samples three strategies: correct modification with probability 0.50, incorrect generation with probability 0.25, and unknown or incomplete generation with probability 0.25.

A correct example changes an attribute to another category-appropriate value and updates the listing consistently. An incorrect example deliberately introduces mismatched evidence. An unknown example removes the relevant attribute information from the text.

Incorrect examples require an additional check. A replacement value can look different while remaining effectively synonymous with the original, producing a weak negative example. The pipeline therefore generates candidate values and uses semantic similarity to select a more distinct replacement.

The broader architecture combines value generation, this semantic-distance check, strategy-conditioned prompting, structural constraints, brand anonymization, and standardized output. The paper’s technical contribution is therefore not simply LLM-generated product prose. It is a procedure for manufacturing defined supervision states from an existing catalog record.

Human review shows high fidelity—and a nontrivial error tail

All 2,000 generated products received three independent expert annotations, with majority vote used for the final judgments.

Reported scores were 99.6% for readability, 96.5% for attribute-value correctness, and 95.8% for brand-modification success. Cross-field consistency was 94.2% for correct examples, 93.0% for incorrect examples, and 88.3% for unknown examples.

The preservation results are more useful for deployment planning. 88.8% of products had no additional changes, 7.0% had acceptable additional changes, and 4.2% contained major unintended changes.

That 4.2% defines a concrete validation burden. A catalog pipeline cannot treat fluency as its release criterion. The relevant failure is an apparently usable record whose non-target information has changed enough to weaken or corrupt the intended label.

The lower consistency score for unknown examples also suggests that removing evidence is harder than simply replacing a value. The paper establishes generation quality for these cases, but not their downstream training value.

The benchmark favors augmentation over replacement

The downstream experiment fine-tunes FLAN-T5-base for attribute extraction under several training-data configurations.

Synthetic-only training nearly matches original-only training. Within this benchmark, the generated records therefore preserve enough task-specific information to function as serious training data, not merely realistic-looking augmentation.

The mixture results are more revealing. Performance peaks at 25% synthetic data and then falls as the synthetic share increases. Synthetic data adds value, but its marginal value is not monotonic.

A production team should therefore treat mixture composition as a parameter to validate, not as a consequence of how cheaply data can be generated. The reported 75/25 split is the best among the configurations tested here; the study does not establish it as a generally optimal ratio across catalogs, languages, architectures, or tasks.

Some apparent model errors are catalog-policy errors

The authors manually examined 441 predictions initially classified as incorrect in the synthetic-only condition. They found cases involving differences in granularity, morphology, units, equivalent definitions, contextual synonyms, multiple valid values, and formatting.

This is an evaluation diagnostic, not evidence that all 441 predictions should be counted as correct. Its operational relevance is elsewhere: attribute extraction performance can be constrained by reference-label and normalization policy as well as by the model.

If two semantically acceptable representations are scored differently, generating more training records will not necessarily resolve the mismatch. Catalog teams may need explicit ontology, unit, synonym, and normalization rules alongside model training.

What catalog teams can use now

The paper directly supports a bounded workflow: start from an existing listing, choose one attribute, define the desired supervision state, constrain unrelated changes, validate the generated record, and then test its contribution to held-out performance.

Cognaptus draws three operational implications from that workflow. First, controlled correct examples can help bootstrap extraction datasets where manual labels are scarce. Second, the incorrect and missing-information generators provide a structured way to create realistic catalog failure cases for future quality workflows. Third, synthetic-data volume should be managed as a training allocation decision rather than maximized automatically.

Only the first implication has downstream training evidence here. The extraction benchmark uses correct examples only. The paper therefore does not establish that its generated incorrect or unknown examples improve learning.

The authors also report two LLM calls per synthetic product and successful regeneration of the complete MAVE dataset as a scalability check. That demonstrates execution at large dataset scale, but production economics would still depend on source cleanup, review requirements, model pricing, and governance.

Where the evidence stops

The study provides credible evidence for controlled synthetic augmentation in one specific setting. It modifies one attribute at a time, evaluates one public product dataset, and uses one downstream model family. Synthetic quality also inherits problems from the source catalog, including vague, missing, or semantically inconsistent attributes.

Most importantly, the benchmark does not justify the conclusion that real catalog data has become dispensable. Synthetic-only training nearly matches original-only performance, while the best tested result retains a large majority of original data.

For catalog operators, that is the more consequential result. The paper turns synthetic generation from a volume question into a supervision-design problem: define exactly what may change, measure what changed unintentionally, and determine experimentally how much generated data improves the training mix.

Cognaptus: Automate the Present, Incubate the Future.