Elamur Research⌕
☰
home / papers / 2013 / Classifying English Documents by National Dialect

Classifying English Documents by National Dialect

Paper · 2013 · Marco Lui and Paul Cook

Summary

Lui and Cook frame national dialect identification as a document classification task: deciding whether an English text is Australian, British or Canadian. They use texts of known national origin as a stand-in for dialect. A classifier trained on one kind of corpus might learn topic, genre or length instead of dialect, so they gather data from several sources: national corpora, web text, government web text and Twitter. Each source is held out in turn. A model trained on the others is then tested on it. They compare methods borrowed from neighbouring fields:

  • text categorization
  • native language identification
  • authorship attribution
  • language identification
  • geolocation
  • spelling-variant ratios

These are scored with macro-averaged F against a random baseline. A linear SVM over bag-of-words transfers best across sources. Topic-free syntactic features (POS plus function-word n-grams) also clearly beat the baseline. This suggests the dialects differ systematically in lexis and syntax, not only in topic. The study is an early, careful setup for separating real dialect signal from corpus artefacts.

Key points

  • The three dialects are Australian, British and Canadian English. American English was left out because suitable resources were surprisingly scarce.
  • There are four source types: NATIONAL, WEB, WEBGOV and TWITTER. Evaluation uses leave-one-source-out training and macro-averaged F-score. The national data covers only 2 of the dialects.
  • TEXT CATEGORIZATION was the strongest method. It is a linear SVM over roughly 3 million bag-of-words terms with no feature selection.
  • NATIVE LID (POS + function-word n-grams with a maximum entropy classifier) came second. AUTHORSHIP ATTRIB (about 400 function words) stayed near the baseline. VARIANT PAIR (spelling-variant ratios) fell below the random baseline across sources.
  • In the single-source transfer matrix, NATIONAL-trained models scored 0.975 on their own data but only 0.181 F on TWITTER (TEXT CATEGORIZATION). Training on all non-test sources generally beat training on any single source.

Interesting ideas

  • I1. Hold out whole data sources rather than random documents, so a classifier only scores well if its signal survives changes of topic, genre and length.
  • I2. When class proportions differ between sources, use a random baseline rather than a majority-class one, because the majority class from training is not meaningful at test time.
  • I3. Use deliberately topic-free features (function words, POS n-grams) to check whether a signal is structural rather than lexical.
  • I4. Build a full train-on-X, test-on-Y matrix across sources to see which data is a poor proxy for which (national corpora transferred badly to Twitter).
  • I5. Spelling-variant ratios work for whole corpora but fail on single documents, because one author rarely mixes variants, so the evidence is too sparse.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate parsers and frame extractors by training on some text genres and testing on an unseen genre (e.g. fables vs. school tests), which exposes genre artefacts.
  • I2 (learn): Use such distribution-independent baselines as negative controls when label priors shift between silver-data sources and evaluation sets.
  • I3 (tree): Run a feature-ablation probe on dialect or genre effects in parser errors to tell syntactic differences apart from vocabulary coverage.
  • I4 (learn): Build the same matrix for treebanks and silver corpora to decide which sources actually help a target domain before mixing them into training.
  • I5 (tokens): Handle colour/color-type variants by lemma normalisation rather than as a per-document dialect cue, and treat any dialect guess from spelling as weak.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Marco Lui and Paul Cook. 2013. Classifying English Documents by National Dialect. Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lui2013classifying,
  title = {{Classifying English Documents by National Dialect}},
  author = {Marco Lui and Paul Cook},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2013 (ALTA 2013)},
  year = {2013},
  url = {https://aclanthology.org/U13-1003/}
}

References

51 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eric Atwell, Junaid Arshad, Chien-Ming Lai, Lan Nim, Noushin Rezapour Asheghi, Josiah Wang, and Justin Washtell. 2007. Which English dominates the World Wide Web, British or American? In Proceedings of the Corpus Linguistics Conference (CL 2007). Birmingham, UK. find
  2. Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrnt social media sources? In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 356–364. Asian Federation of Natural Language Processing, Nagoya, Japan.
  3. Marco Baroni and Silvia Bernardini. 2004. Boot-CaT: Bootstrapping corpora and terms from the Web. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004). find
  4. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky Wide Web: A collection of very large linguistically processed Web-crawled corpora. Language Resources and Evaluation, 43(3):209– 226. ↗
  5. Julian Brooke and Graeme Hirst. 2012. Robust, lexicalized native language identification. In Proceedings of COLING 2012, pages 391– 408. find
  6. The COLING 2012 Organizing Committee, Mumbai, India. find
  7. Lou Burnard. 2000. The British National Corpus Users Reference Guide. Oxford University Computing Services. find
  8. Paul Cook and Graeme Hirst. 2012. Do Web corpora from top-level domains represent national varieties of English? In Proceedings of the 11th International Conference on Textual Data Statistical Analysis, pages 281–293. Liège, Belgium. find
  9. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20:273–297. ↗
  10. Mark Davies. 2009. The 385+ million word Corpus of Contemporary American English (1990– 2008+): Design, architecture, and linguistic insights. International Journal of Corpus Linguistics, 14(2):159–190. ↗
  11. Franca Debole and Fabrizio Sebastiani. 2005. An analysis of the relative hardness of reuters-21578 subsets. Journal of the American Society for Information Science and Technology, 56(6):584–596. ↗
  12. Harris Drucker, Vladimir Vapnik, and Dongui Wu. 1999. Support vector machines for spam categorization. IEEE Transactions on Neural Networks, 10:1048–1054. ↗
  13. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. 2010. A latent variable model for geographic lexical variation. In Empirical Methods in Natural Language Processing, pages 1277–1287. Cambridge, MA, USA.
  14. Dominique Estival, Tanja Gaustad, and Ben Hutchinson. 2007. Author profiling for english emails. In Proccedings of the 10th Conference for the Pacific Association for Computational Linguistics, pages 263–272. Melbourne,Australia. find
  15. Elizabeth Green and Pam Peters. 1991. The Australian corpus project and Australian English. International Computer Archive of Modern English, 15:37–53. find
  16. Bo Han, Paul Cook, and Timothy Baldwin. 2012. Geolocation prediction in social media data by finding location indicative words. In Proceedings of COLING 2012, pages 1045–1062. find
  17. The COLING 2012 Organizing Committee, Mumbai, India. find
  18. Graeme Hirst and Olga Feiguina. 2007. Bigrams of syntactic labels for authorship discrimination of short texts. Literary and Linguistic Computing, 22(4):405–417. ↗
  19. Nancy Ide. 2009. The American National Corpus: Then, now, and tomorrow. In Michael Haugh, editor, Selected Proceedings of the 2008 HC-SNet Workshop on Designing an Australian National Corpus, pages 108–113. Cascadilla Proceedings Project, Sommerville, MA. find
  20. Jennifer Jenkins. 2009. World Englishes: A resource book for students. Routledge, London, second edition. find
  21. Thorsten Joachims. 1998. Text categorization with support vector machines: learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning, pages 137–142. Chemnitz, Germany. ↗
  22. Adam Kilgarriff. 2001. Comparing corpora. International Journal of Corpus Linguistics, 6(1):97–133. find
  23. Adam Kilgarriff, Siva Reddy, Jan Pomikálek, and Avinesh PVS. 2010. A corpus factory for many languages. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC 2010), pages 904–910. Valletta, Malta. find
  24. Moshe Koppel, Jonathan Schler, and Kfir Zigdon. 2005. Automatically determining an anonymous authors native language. Intelligence and Security Informatics, 3495:209–217. find
  25. Marco Lui and Timothy Baldwin. 2011. Cross-domain feature selection for language identification. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), pages 553–561. Chiang Mai, Thailand. find
  26. Marco Lui and Timothy Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012) Demo Session, pages 25–30. Jeju, Republic of Korea.
  27. Frederick Mosteller and David L. Wallace. 1964. Inference and disputed authorship: The Federalist Papers. Addison-Wesley, Reading,USA. find
  28. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. Knowledge and Data Engineering, IEEE Transactions on, 22(10):1345– 1359. ↗
  29. Yves Peirsman, Dirk Geeraerts, and Dirk Speelman. 2010. The automatic identification of lexical variation between language varieties. Natural Language Engineering, 16(4):469–491. ↗
  30. John M. Prager. 1999. Linguini: language identification for multilingual documents. In Proceedings the 32nd Annual Hawaii International Conference on Systems Sciences (HICSS-32). Maui, Hawaii. ↗
  31. John Ross Quinlan. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, USA.
  32. Randolph Quirk. 1995. Grammatical and lexical variance in English. Longman, London. find
  33. Stephen Roller, Michael Speriosu, Sarat Rallapalli, Benjamin Wing, and Jason Baldridge. 2012. Supervised text-based geolocation using language models on an adaptive grid. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1500–1510. Jeju Island, Korea. find
  34. Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia, PA. find
  35. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1–47. ↗
  36. Fabrizio Sebastiani. 2005. Text categorization, pages 109–129. TEMIS Text Mining Solutions S.A., Italy. find
  37. Yanir Seroussi, Fabian Bohnert, and Ingrid Zukerman. 2012. Authorship attribution with author-aware topic models. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 264–269. Association for Computational Linguistics, Jeju Island, Korea. find
  38. Serge Sharoff. 2006. Creating general-purpose corpora using automated search engine queries. In Marco Baroni and Silvia Bernardini, editors, Wacky! Working papers on the Web as Corpus, pages 63–98. GEDIT, Bologna, Italy. find
  39. Efstathios Stamatatos. 2009. A survey of modern authorship attribution methods. Journal of The American Society for Information Science and Technology, 60:538–556. ↗
  40. Dolf Trieschnigg, Djoerd Hiemstra, Mariët Theune, Franciska de Jong, and Theo Meder. 2012. An exploration of language identification techniques for the dutch folktale database. In Proceedings of the LREC workshop on the Adaptation of Language Resources and Tools for Processing Cultural Heritage Objects. find
  41. Peter Trudgill. 1999. Standard English: What it isnt. In Tony Bex and Richard J. Watts, editors, Standard English: The widening debate, pages 117–128. Routledge, London. find
  42. Peter Trudgill and Jean Hannah. 2008. International English: A guide to varieties of Standard English. Hodder Education, London, fifth edition. find
  43. Twitter. 2013. New tweets per second record, and how! https://blog.twitter.com/2013/new-tweets-per-second-record-and-how. trieved 19 August 2013. ↗
  44. Benjamin P. Wing and Jason Baldridge. 2011. Simple supervised document geolocation with geodesic grids. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11, pages 955–964. Association for Computational Linguistics, Portland, Oregon, USA. find
  45. Sze-Meng Jojo Wong and Mark Dras. 2009. Contrastive analysis and native language identification. In Proceedings of the Australasian Language Technology Workshop 2009 (ALTW 2009), pages 53–61. Sydney, Australia. find
  46. Sze-Meng Jojo Wong and Mark Dras. 2011. Exploiting parse structures for native language identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP 2011), pages 1600– 1610. Association for Computational Linguistics, Edinburgh, Scotland, UK. find
  47. Sze-Meng Jojo Wong, Mark Dras, and Mark Johnson. 2012. Exploring adaptor grammars for native language identification. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning 2012 (EMNLP-CoNLL 2012), pages 699–709. Association for Computational Linguistics, Jeju Island, Korea. find
  48. Yiming Yang and Xin Liu. 1999. A re-examination of text categorization methods. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ’99, pages 42–49. ACM Press, New York, USA. find
  49. G. Udny Yule. 1939. On sentence-length as a statistical characteristic of style in prose: With application to two cases of disputed authorship. Biometrika, 30:363–390. ↗
  50. Omar F Zaidan and Chris Callison-Burch. 2012. Arabic dialect identification. Computational Linguistics, 52(1). ↗
  51. Ying Zhao and Justin Zobel. 2005. Effective and Scalable Authorship Attribution Using Function Words. In Asia Information Retrieval Symposium, pages 174–189. Re- find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lui-2013-classifying-english-documents-national-dialect/