Elamur Research⌕
☰
home / papers / 2015 / Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition

Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition

Summary

This paper reports on the two shared tasks organized with the W-NUT 2015 workshop: lexical normalization of English tweets and named entity recognition (NER) on Twitter. Earlier normalization research relied mostly on unsupervised or semi-supervised methods, which were tested on small hand-labelled sets, so it was hard to compare them. To address this, the organizers built a much larger annotated corpus. It covers one-to-one, one-to-many and many-to-one word mappings, and it includes non-standard tokens whose spelling happens to match a real word.

Ten teams took part in normalization. Systems were split into constrained and unconstrained tracks according to which resources they used. Eight teams entered the NER task, which had 10 entity types. The best normalization system reached an F1 of 0.8421 in the constrained track, which suggests that model quality mattered more than extra data. In NER, the ousia system used entity linking and clearly outperformed the other entries. Its score approached the level of agreement between human annotators.

These datasets and results became reference points for processing noisy social-media text.

Key points

  • Normalization corpus:
    • Tweets were collected in May 2014 and filtered with langid.py, then tokenized with the CMU-ARK tokeniser.
    • The final corpus has 4,917 tweets, split into 2,950 for training and 1,967 for testing.
    • Each tweet was labelled by two annotators out of a pool of 12 at IBM Research Australia, and disagreements were adjudicated.
    • Average Cohen's κ was 0.5854.
  • Lexicon:
    • The boundary between standard and non-standard words was set by the SCOWL lexicon, using American spelling at the 70% size level, for 165,458 words in total.
    • Tweets with fewer than two tokens outside this lexicon were discarded so that the annotated data would contain plenty of normalization cases.
  • Normalization results:
    • There were 6 constrained and 5 unconstrained submissions; one team entered both tracks.
    • The winner scored P 0.9061, R 0.7865 and F1 0.8421.
    • Neural networks performed well in the constrained track, while CRF- and lexicon-based systems did well in the unconstrained one.
  • NER results:
    • ousia scored 56.41 F1 with entity types and 70.63 F1 for segmentation only, against baseline scores of 31.97 and 49.88.
    • NLANGP came second with typed F1 51.40.
  • Common NER techniques:
    • Most teams used linear-chain CRFs together with Brown clusters and word embeddings.
    • Less common designs included a semi-Markov MIRA tagger (nrc), a feature-free feed-forward network over word2vec embeddings (multimedialab), and differential-evolution feature selection (iitp).

Interesting ideas

  • I1. Fix the "standard word" boundary with a lexicon: Whether a word counts as standard is a matter of opinion. The organizers avoided endless debate by treating membership in one fixed dictionary as the definition. This makes annotation reproducible and could work for normalizing noisy Ukrainian or Surzhyk text.
  • I2. Pre-filter for annotation density: Only tweets with at least two out-of-lexicon tokens were kept. This means annotators spend their time on informative examples, at the cost of some sampling bias. It is a cheap way to stretch a small annotation budget.
  • I3. Constrained vs. unconstrained tracks: Splitting submissions by resource use shows whether gains come from the model or from extra data. Here, a constrained system won, a useful lesson when choosing between more data and a better model.
  • I4. Use entity linking for NER: ousia linked candidate mentions to a knowledge base. It then used facts about the linked entity, such as its popularity, its classes, and how likely it is to appear in the given context, as classifier features. This approach could help NER in low-resource languages that have a Wikipedia-style knowledge base.
  • I5. Down-weight older training data: USFD gave less weight to training examples that were far in time from the test data, to handle concept drift. This is relevant for any fast-changing social-media or news stream.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-4319

@inproceedings{baldwin2015shared,
  title = {{Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition}},
  author = {Baldwin, Timothy and Marneffe, Marie Catherine de and Han, Bo and Kim, Young-Bum and Ritter, Alan and Xu, Wei},
  booktitle = {Proceedings of the Workshop on Noisy User-generated Text},
  year = {2015},
  doi = {10.18653/v1/w15-4319},
  url = {https://doi.org/10.18653/v1/w15-4319}
}

References

43 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md Shad Akhtar, Utpal Kumar Sikdar, and Asif Ekbal. 2015a. Iitp: Multiobjective differential evolution based twitter named entity recognition. In proceedings of WNUT. find
  2. Md Shad Akhtar, Utpal Kumar Sikdar, and Asif Ekbal. 2015b. Iitp: Hybrid approach for text normalization in twitter. In proceedings of WNUT, Beijing, China. find
  3. AiTi Aw, Min Zhang, Juan Xiao, and Jian Su. 2006. A phrase-based statistical model for SMS text normalization. In Proceedings of COLING/ACL 2006, pages 33–40, Sydney, Australia. find
  4. Russell Beckley. 2015. Bekli:a simple approach to twitter text normalization. In proceedings of WNUT, Beijing, China. find
  5. Gabor Berend and Ervin Tasnádi. 2015. Uszeged: Correction type-sensitive normalization of english tweets using efficiently indexed n-gram statistics. In proceedings of WNUT, Beijing, China. find
  6. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. ACM. find
  7. Colin Cherry and Hongyu Guo. 2015. The unreasonable effectiveness of word representations for twitter named entity recognition. NAACL. find
  8. Colin Cherry, Hongyu Guo, and Chengbi Dai. 2015. Nrc: Infused phrase vectors and updated gazetteers for named entity recognition in twitter. In proceedings of WNUT. find
  9. Grzegorz Chrupała. 2014. Normalizing tweets with edit scripts and recurrent neural embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL 2014), pages 680–686, Baltimore, USA, June. find
  10. Paul Cook and Suzanne Stevenson. 2009. An unsupervised model for text message normalization. In Proceedings of the Workshop on Computational Approaches to Linguistic Creativity (CALC ’09), pages 71–78, Boulder, USA. find
  11. Leon Derczynski, Isabelle Augenstein, and Kalina Bontcheva. 2015a. Usfd: Twitter ner with drift compensation and linked data. In proceedings of WNUT. find
  12. Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015b. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32–49. find
  13. Mark Dredze, Tim Oates, and Christine Piatko. 2010. We’re not in kansas anymore: detecting domain changes in streams. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 585–595. Association for Computational Linguistics. find
  14. Hege Fromreide, Dirk Hovy, and Anders Søgaard. 2014. Crowdsourcing and annotating ner for twitter\# drift. European language resources distribution agency. find
  15. Fréderic Godin, Baptist Vandersmissen, Wesley De Neve, and Rik Van de Walle. 2015. multimedialab @ acl wnut ner shared task: Named entity recognition for twitter microposts using distributed word representations. In proceedings of WNUT. find
  16. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a \#twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL HLT 2011), pages 368–378, Portland, USA. find our card
  17. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning 2012 (EMNLP-CoNLL 2012), pages 421– 432, Jeju Isl… find
  18. Bo Han, Paul Cook, and Timothy Baldwin. 2013. Lexical normalisation for social media text. ACM Transactions on Intelligent Systems and Technology, 4(1):5:1–5:27. find
  19. Hany Hassan and Arul Menezes. 2013. Social text normalization using contextual graph random walks. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), pages 1577–1586, Sofia, Bulgaria, August. find
  20. Ning Jin. 2015. Ncsu-sas-ning: Candidate generation and feature engineering for supervised lexical normalization. In proceedings of WNUT, Beijing, China. find
  21. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and A. Noah Smith. 2014. A dependency parser for tweets. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1001–1012. find our card
  22. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning. find
  23. Samuel Leeman-Munk, James Lester, and James Cox. 2015. Ncsu sas sam: Deep encoding and reconstruction for normalization of noisy text. In proceedings of WNUT, Beijing, China. find
  24. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2013. Paraphrasing 4 microblog normalization. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pages 73–84, Seattle, USA, October. find
  25. Fei Liu, Fuliang Weng, Bingqing Wang, and Yang Liu. 2011. Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL HLT 2011)… find
  26. Fei Liu, Fuliang Weng, and Xiao Jiang. 2012. A broad-coverage normalization system for social media language. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), pages 1035–1044, Jeju Island, Korea, July. find
  27. Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012) Demo Session, pages 25–30, Jeju, Republic of Korea. find our card
  28. Wookhee Min, Bradford Mott, James Lester, and James Cox. 2015. Ncsu sas wookhee: A deep contextual long-short term memory model for text normalization. In proceedings of WNUT, Beijing, China. find
  29. Yerai Doval Mosquera, Jesús Vilares, and Carlos Gómez-Rodrı́guez. 2015. Lysgroup: Adapting a spanish microtext normalization system to english. In proceedings of WNUT, Beijing, China. find
  30. Barbara Plank, Dirk Hovy, Ryan McDonald, and Anders Søgaard. 2014a. Adapting taggers to twitter with not-so-distant supervision. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1783–1792. find
  31. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014b. Learning part-of-speech taggers with inter-annotator agreement loss. In Proceedings of EACL. find our card
  32. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1524–1534. Association for Computational Linguistics. find our card
  33. Richard Sproat, Alan W. Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christopher Richards. 2001. Normalization of non-standard words. Computer Speech and Language, 15(3):287–333. find
  34. Dmitry Supranovich and Viachaslau Patsepnia. 2015. Ihs rd: Lexical normalization for english tweets. In proceedings of WNUT, Beijing, China. find
  35. Tian Tian. 2015. Data adaptation for named entity recognition on tweets with features-rich crf. In proceedings of WNUT. find
  36. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. find
  37. Zhiqiang Toh, Bin Chen, and Jian Su. 2015. Improving twitter named entity recognition using word representations. In proceedings of WNUT. find
  38. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics. find our card
  39. Joachim Wagner and Jennifer Foster. 2015. Dcu-adapt: Learning edit operations for microblog normalisation with the generalised perceptron. In proceedings of WNUT, Beijing, China. find
  40. Pidong Wang and Hwee Tou Ng. 2013. A beam-search decoder for normalization of social media text with application to machine translation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT 2013),… find
  41. Ikuya Yamada, Hideaki Takeda, and Takefuji Yoshiyasu. 2015. Enhancing named entity recognition in twitter messages using entity linking. In proceedings of WNUT. find
  42. Yi Yang and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pages 61–72, Seattle, USA, October. find
  43. Eun-Suk Yang and Yu-Seop Kim. 2015. Hallym: Named entity recognition on twitter. In proceedings of WNUT. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baldwin-2015-shared-tasks-2015-workshop-noisy/