Informal Text Breaks G2P Before Pronunciation Starts

TL;DR for operators

A grapheme-to-phoneme system can look accurate on ordinary words and deteriorate sharply when users write abbreviations, slang, lengthenings, or phonetic spellings. In UGTPhon, the largest observed difference between canonical and non-canonical inputs is 66.8 phoneme-error-rate points.

The paper’s more consequential result is diagnostic. Some informal forms preserve enough pronunciation information to be read almost directly. Others omit information that must first be recovered from context or prior knowledge. The proposed system therefore keeps sentence context, retrieves an exact-match canonical-form hint when available, and for non-canonical inputs explicitly generates the intended standard form before generating phonemes.

Against matched ByT5 and Qwen2.5-0.5B baselines, that formulation lowers non-canonical phoneme error in all three evaluated languages. The largest gains occur where reconstruction is most necessary. More UGT fine-tuning alone does not remove the problem, and a separate normalization stage can reduce informal-text error while degrading ordinary-word accuracy.

For production voice systems, the immediate decision is therefore not simply whether to add more training data. Measure canonical and non-canonical traffic separately, classify the failures by reconstruction demand, and monitor retrieval coverage if a datastore is used. The evidence covers English, Vietnamese, and Korean; it does not establish general robustness across writing systems or perceived TTS quality.

Informal spelling can remove information needed for pronunciation

A voice product has to pronounce ordinary text and user-generated forms through the same front end. That includes inputs such as pls, soooo, luv, abbreviations, slang, and other spellings that may never appear in a conventional pronunciation lexicon.

The difficulty is not uniform. Some altered spellings still contain most of what a pronunciation model needs. Others hide part of the intended word or phrase. In those cases, generating the right sounds requires recovering information that is absent from the surface form.

That distinction becomes visible in Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach.¹ Across the evaluated systems, the largest gap between error on ordinary canonical text and non-canonical user-generated text reaches 66.8 phoneme-error-rate points. Phoneme error rate, or PER, measures errors in the generated phoneme sequence; the paper reports the robustness gap as

A large positive $\Delta$ means a system that performs adequately on standard words can fail much more often once the spelling changes.

UGTPhon measures the inference required before pronunciation

UGTPhon contains 18,373 sentences: 5,029 English, 10,847 Vietnamese, and 2,497 Korean. Evaluation is performed on a marked token or span while the model can condition on the surrounding sentence.

Its useful design choice is the taxonomy. Instead of grouping examples only by what changed visually, the benchmark asks what the model must infer.

Reconstruct cases omit pronunciation-relevant content. pls → please is an English example: the intended standard form contains information that the observed string does not fully express.

Repetition cases involve structures such as character lengthening or repeated words, for example soooo → so.

Pass-through cases alter orthography while retaining comparatively more pronunciation evidence, such as luv → love.

This matters because aggregate “informal text” accuracy mixes different problems. The paper finds the largest improvements from explicit canonical-form modeling in Reconstruct categories, where the missing intermediate representation is most consequential.

More exposure reduces error, but it does not remove the gap

Further supervised training on UGTPhon improves several systems, particularly in English and Vietnamese. It does not make canonical and non-canonical performance converge.

For the 300M-parameter ByT5 model after further UGTPhon training, non-canonical PER remains 29.8 in English, 43.2 in Vietnamese, and 47.5 in Korean. Korean ByT5 does not improve relative to its canonical-lexicon-trained version on this metric.

The paper’s main matched comparisons test a different intervention: preserve sentence context, optionally retrieve the intended canonical form from a training-derived exact-match datastore, and make that canonical form an explicit intermediate output before phoneme generation.

These are matched-backbone comparisons, so they provide stronger evidence about the formulation than comparisons against unrelated model families. Two-sided paired-bootstrap 95% confidence intervals exclude zero for all six comparisons. The English Qwen reduction is statistically separated from zero but numerically small: 0.5 PER points.

The Korean Qwen result is much larger, but Korean also has the smallest test split, substantially lower training-datastore coverage, and higher reported multi-seed variance. Its magnitude should therefore not be treated as an expected gain for another language.

The reconstruction step works best when the input actually needs reconstruction

The component experiments help explain where the improvement comes from.

Sentence context alone is useful when the system still predicts only the marked target. Generating an entire output sentence, by contrast, sharply damages canonical performance in the reported ablation. The distinction is architectural: context can supply evidence without forcing the model to regenerate material irrelevant to the target pronunciation.

Retrieval contributes another source of evidence. If a non-canonical target appears in the datastore, the model receives its surface-to-canonical mapping; otherwise it receives an explicit miss marker and must rely on context.

For non-canonical inputs, staged decoding then trains the system to emit the intended canonical form $G_c$ before the phoneme sequence $P_c$:

Canonical inputs bypass that reconstruction step and emit phonemes directly.

In the ByT5 ablation, retrieval and staged decoding are complementary in English and Vietnamese, although their contributions are not uniform across languages. Korean illustrates that boundary: retrieval alone produces a lower non-canonical PER than retrieval plus staged decoding in that particular ablation. The evidence supports explicit reconstruction as a useful mechanism, not a claim that every component improves every setting monotonically.

A separate normalizer can create a second failure mode

An upstream lexical normalizer appears attractive because it separates informal-text handling from pronunciation. The paper tests that family of pipelines and finds a trade-off: normalization can lower error on non-canonical inputs while substantially degrading canonical-word accuracy.

The proposed formulation keeps reconstruction conditional. Canonical targets go directly to phoneme generation, while non-canonical targets can use the intermediate canonical representation.

For a production team, that changes the evaluation requirement. Improving slang handling is insufficient if the intervention starts “correcting” ordinary words that were already handled properly. Canonical regression should be a release criterion alongside non-canonical improvement.

Simple retrieval captures much of the observed deployable gain

The retrieval analysis is also operationally relevant. With ByT5, a datastore built only from training examples yields non-canonical PER of 22.7 in English and 26.7 in Vietnamese, versus oracle values of 20.6 and 25.9. Adding LLM-generated datastore expansions changes those results only slightly, to 22.5 and 26.7.

That experiment is a sensitivity test rather than a second main result, but it suggests that a relatively simple datastore can capture much of the retrieval benefit observed in English and Vietnamese.

Coverage is the constraint. The training-only datastore covers 70.9% of English, 82.9% of Vietnamese, and only 25.3% of Korean non-canonical test-target occurrences. User-generated vocabulary also changes continuously.

Cognaptus therefore infers that a retrieval-backed production system needs coverage monitoring and a governed refresh process. A useful dashboard would report at least non-canonical error, retrieval hit rate, unseen-form rate, and performance by reconstruction category. The paper itself does not test such a production monitoring system.

What the evidence changes for voice-system decisions

The paper directly supports three decisions.

First, evaluate informal text as its own slice. Aggregate G2P accuracy can conceal a large robustness gap.

Second, diagnose by inference demand rather than spelling appearance alone. Reconstruct cases are materially different from repetition or pass-through cases because the pronunciation system may need information that the observed token does not contain.

Third, test canonical regression whenever adding normalization machinery. A method that improves slang while degrading standard text has moved the error rather than resolved the deployment problem.

The broader business inference is narrower than “add retrieval.” Retrieval is attractive when recurring surface forms account for enough traffic to justify maintaining a mapping datastore. Where unseen forms dominate, another source of contextual or learned reconstruction may be required.

The benchmark does not establish universal speech robustness

UGTPhon covers three languages and does not test logographic, abjad, or abugida writing systems. The results therefore do not establish that the same taxonomy, retrieval coverage, or effect sizes transfer across script families.

The downstream TTS-ASR experiments in English and Vietnamese move in the same direction as the token-level PER results, but the study does not include subjective mean-opinion-score ratings or multiple TTS systems. Lower PER is therefore evidence of better phonemization under this setup, not a complete measure of perceived speech quality.

The benchmark also inherits toxic material from source corpora and language-specific licensing constraints. Those issues affect whether and how the dataset can be incorporated into commercial data pipelines.

The central result is more precise: informal text creates distinct pronunciation failures when the written form no longer contains everything needed to recover the intended word. Once that intermediate reconstruction requirement is measured explicitly, model design and evaluation can target the failure rather than treating all unconventional spelling as the same kind of noise.

Cognaptus: Automate the Present, Incubate the Future.