Elamur Research⌕
☰
home / papers / 2011 / Lexical Normalisation of Short Text Messages: Makn Sens a #twitter

Lexical Normalisation of Short Text Messages: Makn Sens a #twitter

Summary

The paper addresses lexical normalisation of noisy English short messages: mapping out-of-vocabulary (OOV) tokens such as typos, phonetic spellings (b4, 2morrow) and lengthened words (goooood) to a single standard word, so that downstream tools like parsers work better. A pilot study shows Twitter has many more OOV tokens per message than edited news, and the non-standard forms vary widely. Because of this long tail, supervised learning would suffer from sparse data. The authors therefore propose a cascade that needs no annotated training data. A classifier first decides whether an OOV token is ill-formed. Candidate corrections are then generated from morphological and phonetic similarity. Finally, candidates are ranked using word-similarity and context features, with a slang dictionary consulted first. The authors also build a new Twitter normalisation dataset. Evaluated on an SMS corpus and on this Twitter set, the full combination beats a noisy channel baseline and an SMT baseline in both F-score and BLEU. This shows that context-aware, annotation-free normalisation can be practical for social media text.

Key points

  • OOV status is decided with the GNU aspell dictionary (v0.60.6). Slang lookup uses a 5021-entry list gathered from noslang.com. Mentions, hashtags and URLs are not normalised but are kept as context.
  • In the pilot study, about 15% of tweets have half or more of their tokens OOV. NYT text from English Gigaword shows a much more Zipfian pattern.
  • The authors manually analysed 449 tweets and found 254 tokens needing normalisation:
    • simple letter insertion/omission accounts for 72.44%;
    • slang accounts for 12.20%;
    • number substitution accounts for 2.76%;
    • mixed letter-and-number forms account for 2.36%.
  • The best configuration, DL+WS+CS, reaches F-score 0.755 and BLEU 0.876 on SMS, and F-score 0.753 and BLEU 0.934 on Twitter. By comparison, Cook and Stevenson's (2009) noisy-channel approach achieves roughly 0.46 F on each dataset.
  • Dictionary lookup alone has the highest precision (0.961 on Twitter) but weak recall (0.460). Context support alone is weak, yet adding it to word similarity gives a statistically significant gain (p < 0.05). Only 32.6% of ill-formed words had context windows made up entirely of IV words.

Interesting ideas

  • I1. Narrow, single-token task definition. The task only covers OOV tokens with one-word targets, so multi-word expansions like imo are out of scope. This keeps the problem tractable and evaluation clean, and the same scoping approach could fit normalisation for other languages, such as Ukrainian social media.
  • I2. Profiling OOV density per message. Binning messages by their share of OOV tokens and comparing the distributions across genres is a cheap diagnostic. It shows how noisy a domain is and whether surrounding context can be trusted before you design a model.
  • I3. High-precision lookup first, then a general model. A slang dictionary handles frequent shorthands, and the similarity-plus-context ranker handles everything else. This is a good pattern wherever a small curated lexicon exists but cannot cover the long tail.
  • I4. Token-level rather than type-level decisions. The paper notes that the same form can need different corrections in different contexts, for example hw meaning "how" in one message and "homework" in another. Context-sensitive, per-occurrence normalisation therefore beats a fixed mapping table.
  • I5. Keep non-normalisable tokens as context. Mentions, hashtags and URLs are left in place rather than deleted, so they can still inform context modelling. This is useful in any pipeline that would otherwise strip such tokens and lose signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

145 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 132 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{han2011lexical,
  title = {{Lexical Normalisation of Short Text Messages: Makn Sens a \#twitter}},
  author = {Han, Bo and Baldwin, Timothy},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1038/}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AiTi Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for SMS text normalization. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 33–40, Sydney, Australia. find
  2. Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. In HLT ’10: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 229–237, Los Angeles, USA. find
  3. Richard Beaufort, Sophie Roekhaut, Louise-Amélie Cougnon, and Cédrick Fairon. 2010. A hybrid rule/model-based finite-state framework for normalizing SMS messages. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 770–779, Uppsala, Sweden. find
  4. Eric Brill and Robert C. Moore. 2000. An improved error model for noisy channel spelling correction. In ACL ’00: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 286–293, Hong Kong. find our card
  5. Kevin Burton, Akshay Java, and Ian Soboroff. 2009. find
  6. The ICWSM 2009 Spinn3r Dataset. In Proceedings of the Third Annual Conference on Weblogs and Social Media, San Jose, USA. find
  7. Monojit Choudhury, Rahul Saraf, Vijit Jain, Animesh Mukherjee, Sudeshna Sarkar, and Anupam Basu. 2007. Investigation and modeling of the structure of texting language. International Journal on Document Analysis and Recognition, 10:157–174. find
  8. Paul Cook and Suzanne Stevenson. 2009. An unsupervised model for text message normalization. In CALC ’09: Proceedings of the Workshop on Computational Approaches to Linguistic Creativity, pages 71– 78, Boulder, USA. find
  9. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy. find our card
  10. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research, 9:1871–1874. find
  11. Graeme Hirst and Alexander Budanitsky. 2005. Correcting real-word spelling errors by restoring lexical cohesion. Natural Language Engineering, 11:87–111. find
  12. Yijue How and Min-Yen Kan. 2005. Optimizing predictive text entry for short message service on mobile phones. In Human Computer Interfaces International (HCII 05), Las Vegas, USA. find
  13. Emi Izumi, Kiyotaka Uchimoto, Toyomi Saiga, Thepchai Supnithi, and Hitoshi Isahara. 2003. Automatic error detection in the Japanese learners’ English spoken data. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 2, pages 145–148, Sapporo, Japan. find
  14. Joseph Kaufmann and Jugal Kalita. 2010. Syntactic normalization of Twitter messages. In International Conference on Natural Language Processing, Kharagpur, India. find
  15. Dan Klein and Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS 2002), pages 3–10, Whistler, Canada. find
  16. Catherine Kobus, Franois Yvon, and Graldine Damnati. 2008. Transcrire les SMS comme on reconnat la parole. In Actes de la Confrence sur le Traitement Automatique des Langues (TALN’08), pages 128–138. find
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: open source toolkit for statistical machine translation. In Pro… find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, USA. find our card
  19. James L. Peterson. 1980. Computer programs for detecting and correcting spelling errors. Commun. ACM, 23:676–687, December. find
  20. Lawrence Philips. 2000. The double metaphone search algorithm. C/C++ Users Journal, 18:38–43. find
  21. Lawrence R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286. find
  22. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In HLT ’10: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 172–180, Los Angeles, USA. find our card
  23. Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379–423, 623–656. find
  24. Richard Sproat, Alan W. Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christopher Richards. 2001. Normalization of non-standard words. Computer Speech and Language, 15(3):287 – 333. find
  25. Andreas Stolcke. 2002. Srilm - an extensible language modeling toolkit. In International Conference on Spoken Language Processing, pages 901–904, Denver, USA. find
  26. Guihua Sun, Gao Cong, Xiaohua Liu, Chin-Yew Lin, and Ming Zhou. 2007. Mining sequential patterns and tree patterns to detect erroneous sentences. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 81–88, Prague, Czech Republic. find
  27. Kristina Toutanova and Robert C. Moore. 2002. Pronunciation modeling for improved spelling correction. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 144–151, Philadelphia, USA. find
  28. Twitter. 2010. Big goals, big game, big records. http://blog.twitter.com/2010/06/big-goals-big-game-big-records.html. Retrieved 4 August 2010. ↗
  29. Wilson Wong, Wei Liu, and Mohammed Bennamoun. 2006. Integrated scoring for spelling error correction, abbreviation expansion and case restoration in dirty text. In Proceedings of the Fifth Australasian Conference on Data Mining and Analytics, pages 83–89, Sydney, Australia. find
  30. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th Conference on Computational Linguistics - Volume 2, COLING ’00, pages 947–953, Saarbrücken, Germany. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/han-2011-lexical-normalisation-short-text-messages/