Elamur Research⌕
☰
home / papers / 2018 / The University of Texas System Submission for the Code-Switching Workshop Shared Task 2018

The University of Texas System Submission for the Code-Switching Workshop Shared Task 2018

Paper · 2018 · Florian Janke, Tongrui Li, Eric Rincón et al.

Summary

This paper describes the BATs group's entry to the named entity recognition shared task at CALCS 2018. The task was to find and classify names in code-switched Twitter posts. Mixed-language tweets are difficult for several reasons. A name may take different forms in each language, be shared across both, or match an ordinary noun in the other language. Tweets also often lack reliable spelling and capitalization. The authors used the roughly 62k annotated tweets, labelled with ten entity types in IOB format. On them they trained a conditional random field with hand-crafted features: word form and suffixes, capitalization, NLTK POS and NE tags, tweet-level language probabilities from langdetect, and Twitter cues such as URLs, mentions, hashtags and emoji, plus features of neighbouring words. The CRF reached weighted F-scores of 65.62 on MSA-EGY and 54.16 on ENG-SPA, beating the baseline. A TensorFlow wide-and-deep network built on fewer features did much worse, mainly because its recall was very low. Later ablations showed that a feature's value depends on the language pair, which is a useful caution for anyone designing features for several languages at once.

Key points

  • The data were about 62k tweets annotated for the CALCS 2018 NER task. Tagging followed the IOB scheme over ten classes: eight entity types (TIME, EVENT, PERSON, GROUP, TITLE, PRODUCT, LOCATION, ORGANIZATION) plus OTHER and NOT-NE, and @-mentions and hashtags always received the O label.
  • The CRF was built with python-crfsuite (a wrapper over CRFsuite), using L1 = 1.0, L2 = 0.001 and 150 training iterations. Features fell into five groups: word, capitalization, NLTK tags, langdetect language probabilities (ENG-SPA only) and Twitter-specific cues.
  • Official submission scores were 65.62 on MSA-EGY and 54.16 on ENG-SPA. Both pairs scored above the baseline, and MSA-EGY scored higher despite the difference in data size.
  • Post-deadline ablations used scikit-learn F1 on the test set:
    • ENG-SPA was best (F1 0.34) when POS/NE tags were dropped or an ASCII check was added.
    • MSA-EGY was best (0.77) with a combined "special" configuration.
    • The ASCII feature helped ENG-SPA but hurt MSA-EGY.
  • The wide-and-deep model (DNNLinearCombinedClassifier) reached only about 0.03–0.07 F1 on ENG-SPA, and the wide-only variant beat the deep+wide one. Deep training was roughly 20 times slower per iteration, so the team discarded the network.

Interesting ideas

  • I1. Running the same feature toggles on two language pairs showed features that help one dataset and hurt the other, so a single global feature set can mislead.
  • I2. Outputs of ready-made NLTK taggers (POS, NE) were added as input features to a sequence model, letting a weak external tool act as one signal among many.
  • I3. A whole-tweet language probability was attached to every token, falling back to zero when the detector guessed an unrelated language. This bounds the damage from detector errors.
  • I4. With few features, a plain linear model beat the deep network and trained about twenty times faster, so model capacity cannot make up for missing features.
  • I5. The annotation scheme decided in advance that @-mentions and hashtags are never entities, removing a whole class of ambiguous decisions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could report ablations separately per treebank or text genre, flagging repair rules or features whose sign flips between them.
  • I2 (tokens): the en tagger could take tags proposed by other taggers as extra perceptron features, with each one's weight visible for explanation.
  • I3 (tokens): document-level signals such as register or dialect could be passed to token features softly, with a neutral default when the detector is unsure.
  • I4 (tree): Mova should keep the perceptron parser as a cheap reference whenever gparse is retrained, to catch cases where the transformer underperforms for lack of useful input.
  • I5 (world): Mova could fix explicit rules for which token types are allowed to become world entities, so entity creation stays predictable and explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States119 works · 57,950 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works

Cite

Cite as Florian Janke, Tongrui Li, Eric Rincón, Gualberto Guzmán, Barbara Bullock and Almeida Jacqueline Toribio. 2018. The University of Texas System Submission for the Code-Switching Workshop Shared Task 2018. Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-3216

@inproceedings{janke2018university,
  title = {{The University of Texas System Submission for the Code-Switching Workshop Shared Task 2018}},
  author = {Florian Janke, Tongrui Li, Eric Rincón, Gualberto Guzmán, Barbara Bullock and Almeida Jacqueline Toribio},
  booktitle = {Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching},
  year = {2018},
  doi = {10.18653/v1/w18-3216},
  url = {https://doi.org/10.18653/v1/w18-3216}
}

References

24 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anupam Jamatia, Björn Gambäck, and Amitava Das. 2015. Part-of-speech tagging for code-mixed english-hindi twitter and facebook chat messages. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 239–248. find
  2. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265– 283. ↗
  3. Anupam Jamatia, Björn Gambäck, and Amitava Das. 2016. Collecting and annotating indian social media code-mixed corpora. In the 17th International Conference on Intelligent Text Processing and Computational Linguistics, pages 3–9. find
  4. Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2018. Overview of the CALCS 2018 Shared Task: Named Entity Recognition on Code-switched Data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, Melbourne, Australia. Association for Computational Linguistics. find
  5. David Jurgens, Stefan Dimitrov, and Derek Ruths. 2014. Twitter users# codeswitch hashtags!# moltoimportante# wow. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 51–61. find
  6. Steven Bird and Edward Loper. 2004. Nltk: the natural language toolkit. In Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, page 31. Association for Computational Linguistics.
  7. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  8. Chu-Cheng Lin, Waleed Ammar, Lori Levin, and Chris Dyer. 2014. The cmu submission for the shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 80–86. find
  9. Özlem Çetinoglu. 2016. A turkish-german code-switching corpus. In LREC. find
  10. Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS 2016, pages 7–10, New York, NY, USA. ACM. find
  11. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. ↗
  12. Shuyo Nakatani. 2010. Language detection library for java. find
  13. Naoaki Okazaki. 2007. Crfsuite: a fast implementation of conditional random fields (crfs). find
  14. Gokul Chittaranjan, Yogarshi Vyas, Kalika Bali, and Monojit Choudhury. 2014. Word-level language identification using crf: Code-switching shared task report of msr india system. In Proceedings of The First Workshop on Computational Approaches to Code Switching, pages 73–79. find
  15. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. ↗
  16. Michal Danilák. 2017. Python port of google language detection library. find
  17. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147. Association for Computational Linguistics.
  18. Terry Peng and Mikhail Korobov. 2014. python-crfsuite. https://python-crfsuite.readthedocs.org/. ↗
  19. Kashif Riaz. 2010. Rule-based named entity recognition in urdu. In Proceedings of the 2010 named entities workshop, pages 126–135. Association for Computational Linguistics. find
  20. Gualberto Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. Proc. Interspeech 2017, pages 67–71. find
  21. Sunayana Sitaram and Alan W Black. 2016. Speech synthesis of code-mixed text. In LREC. find
  22. David Vilares, Miguel A Alonso, and Carlos Gómez-Rodrı́guez. 2015. Sentiment analysis on monolingual, multilingual and code-switching twitter corpora. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2–8. find
  23. Gualberto A Guzman, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2016. Simple tools for exploring variation in code-switching for linguists. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 12–20. find
  24. Ayah Zirikly and Mona Diab. 2015. Named entity recognition for arabic social media. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 176–185.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/janke-2018-university-texas-system-submission-code/