📄 IHS_RD: Lexical Normalization for English Tweets

Summary

The paper addresses lexical normalization of English tweets: rewriting slang, typos, abbreviations and stretched spellings into canonical English for the ACL 2015 W-NUT shared task. Normalizations can be one-to-one, many-to-one or one-to-many, and proper nouns and hashtags/mentions must be left alone. The training data has 2950 annotated tweets.

The authors split the task into two parts:

• Detection: a CRF labels each token as a likely normalization target or not.
• Replacement: a lexicon supplies the rewrite. The lexicon combines the training pairs, earlier normalization datasets (Han; Liu) and a public list of social-media abbreviations.

All output is then converted to American spelling with VarCon. An in-house SVM "did-you-mean" spelling corrector is applied afterwards to catch words the lexicon lacks.

On the 1967-tweet test set the submitted system scored F1 0.8272. It ranked first of 5 unconstrained entries and second of all 11. Dropping the corrector actually raised F1 slightly, because it overcorrected non-English and morphologically varied words. The study shows that a simple, transparent pipeline built around a lexicon can be highly competitive.

Key points

• Candidate CRF:
  • Labels are CAND / NOT_CAND.
  • Features are the token itself, its left and right neighbours, and compliance with the guidelines (alphanumeric only).
  • Membership in the normalization lexicon and in the canonical lexicon are also features.
  • So are word length, vowel count and their interaction, plus whether the token is within edit distance ≤2 of a canonical word.
• Effect of features (CRF stage alone):
  • The baseline features alone give CRF precision 0.991 but recall only 0.57 (F1 0.7237).
  • Adding the dictionary features lifts CRF F1 to 0.8635.
  • With all features enabled, CRF F1 is 0.8632, and the final system F1 is 0.8272.
• With vs. without the corrector:
  • Submitted system with DYM: P 0.8469, R 0.8083, F1 0.8272.
  • Without DYM: P 0.8765, R 0.7949, F1 0.8337.
• The DYM corrector:
  • It is an SVM over candidate corrections.
  • Its features include an error-model score from an autocorrection index of 12.4M documents and a 2-gram language model trained on 177K patents.
  • It was not adapted to social media text.
• Approaches that failed:
  • Language detection did not help, because tweets are short and often mix languages.
  • A non-social-media language model could not pick the right variant among ambiguous entries (e.g. ur → you're / your / you).
  • As a fallback, the most frequent variant is listed first.

Interesting ideas

• I1. Separate "does this token need changing?" (a sequence classifier) from "what does it become?" (a lexicon lookup); each half then stays simple, inspectable and fixable on its own.
• I2. A cheap length-versus-vowel-count feature catches vowel-stripped abbreviations like bcz or pls without any dictionary.
• I3. Score each component on its own and also end-to-end; here a slightly better candidate detector did not translate into a much better final result, and an extra corrector lowered overall F1.
• I4. The corrector treats "make no correction" as an explicit candidate with its own confidence, rather than as the absence of an output.
• I5. A corrector trained on patents and scientific text overcorrected foreign and inflected words in tweets, which shows the danger of applying a module to text from a different domain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): a pre-tagger normalization pass in crate en could flag noisy tokens with a classifier and rewrite them only from an explicit lexicon, so every change is traceable to an entry.
• I2 (tokens): the tokenizer/tagger could use this signal to route odd tokens to an abbreviation-expansion step instead of guessing a POS for an unknown word.
• I3 (learn): evaluation should report per-stage metrics next to the final answer score, so that a module that looks good locally but harms the pipeline is visible.
• I4 (learn): self-induced repair rules could always compete against a scored "leave unchanged" option, which serves as a built-in guard against overcorrection.
• I5 (learn): when silver data or repair modules are reused on a new genre, run them first against held-out text from that genre as a negative control before trusting their changes.