Elamur Research⌕
☰
home / papers / 2014 / A Large-Scale Pseudoword-Based Evaluation Framework for State-of-the-Art Word Sense Disambiguation

A Large-Scale Pseudoword-Based Evaluation Framework for State-of-the-Art Word Sense Disambiguation

Paper · 2014 · Mohammad Taher Pilehvar and Roberto Navigli

Summary

Word Sense Disambiguation (WSD) evaluation usually relies on small, manually annotated test sets. These are too limited to show which factors drive system accuracy. Pilehvar and Navigli address this by simulating evaluation at scale with pseudowords: artificial ambiguous words built by merging unambiguous words that stand in for the senses of a real polysemous word.

The authors introduce two ways to build "semantically aware" pseudowords with wide coverage. The first, similarity-based, runs Personalized PageRank over WordNet to find the closest monosemous proxy for each sense. The second relies on topic signatures. They check how realistic these pseudowords are, then use the best variant to produce pseudosense-tagged corpora of millions of sentences, restricted to nouns.

On this framework they compare the supervised system IMS with the knowledge-based system UKB, varying the amount of training data and of knowledge. IMS comes out ahead overall, and its advantage widens at higher polysemy when sense distributions are natural. The framework makes fine-grained analysis of WSD paradigms possible without costly manual annotation.

Key points

  • Two pseudoword generators: a similarity-based method (Personalized PageRank on the WordNet graph, from the authors' 2013 work) and a new topic-signature-based method. Both aim to mimic real words in disambiguation difficulty and sense distinguishability.
  • Systems and configurations: IMS (SVM-based, supervised) is compared with UKB (PPR-based, knowledge-based) under four combinations of uniform and natural pseudosense distributions: Uni-Uni, Uni-Nat, Nat-Uni and Nat-Nat. Training sets range up to 800 sentences per pseudoword.
  • Effect of polysemy: in Nat-Nat, the IMS lead grows from 13 points at polysemy 2 to 35.9 at polysemy 10, averaging 28.2. In the other three configurations the gap averages about 14.1.
  • Graph connectivity: UKB recall rises with the degree of a pseudosense node in the enriched lexical knowledge base, which is WordNet plus co-occurrence-derived edges. This trend holds at both 80 and 800 training sentences.
  • Motivation: available IMS models supply training examples for only about one third of ambiguous WordNet nouns, which illustrates the knowledge acquisition bottleneck.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy57 works · 9,564 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksWord sense disambiguation · 83 worksLexical semantics & word embeddings · 63 works

Cite

Cite as Mohammad Taher Pilehvar and Roberto Navigli. 2014. A Large-Scale Pseudoword-Based Evaluation Framework for State-of-the-Art Word Sense Disambiguation. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00202

@article{pilehvar2014large,
  title = {{A Large-Scale Pseudoword-Based Evaluation Framework for State-of-the-Art Word Sense Disambiguation}},
  author = {Mohammad Taher Pilehvar and Roberto Navigli},
  journal = {Computational Linguistics},
  year = {2014},
  doi = {10.1162/coli\_a\_00202},
  url = {https://doi.org/10.1162/coli_a_00202}
}

References

89 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, Eneko, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of Human Language Technologies 2009: The Conference of the North American Chapter of the Associa…
  2. Agirre, Eneko, Olatz Ansa, David Martinez, and Eduard Hovy. 2001. Enriching WordNet concepts with topic signatures. In Proceedings of the NAACL Workshop on WordNet and Other Lexical Resources, pages 23–28, Pittsburg, PA. find
  3. Agirre, Eneko and Oier Lopez de Lacalle. 2004. Publicly available topic signatures for all WordNet nominal senses. In Proceedings of the LREC, pages 1,123–1,126, Lisbon. find
  4. Agirre, Eneko, Oier Lopez de Lacalle, and Aitor Soroa. 2009. Knowledge-based WSD on specific domains: Performing better than generic supervised WSD. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI), pages 1,501–1,506, Pasadena, CA. find
  5. Agirre, Eneko, Oier Lopez de Lacalle, and Aitor Soroa. 2014. Random walks for knowledge-based Word Sense Disambiguation. Computational Linguistics, 40(1):57–84.
  6. Agirre, Eneko and David Martı́nez. 2004. Unsupervised WSD based on automatically retrieved examples: The importance of bias. In Proceedings of EMNLP, pages 25–32, Barcelona. find
  7. Agirre, Eneko and Aitor Soroa. 2009. Personalizing PageRank for Word Sense Disambiguation. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 33–41, Athens. find
  8. Banko, Michele and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 26–33, Toulouse.
  9. Bergsma, Shane, Dekang Lin, and Randy Goebel. 2008. Discriminative learning of selectional preference from unlabeled text. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 59–68, Honolulu, HI. find
  10. Bordag, Stefan. 2006. Word Sense Induction: Triplet-based clustering and automatic evaluation. In Proceedings of the 11 th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 137–144, Trento. find
  11. Buchanan, Bruce G. and David C. Wilkins, editors. 1993. Readings in Knowledge Acquisition and Learning: Automating the Construction and Improvement of Expert Systems. Morgan Kaufmann Publishers Inc., San Francisco, CA. find
  12. Chambers, Nathanael and Dan Jurafsky. 2010. Improving the use of pseudo-words for evaluating Selectional Preferences. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 445–453, Uppsala. find
  13. Chan, Yee Seng and Hwee Tou Ng. 2005a. Scaling up word sense disambiguation via parallel texts. In Proceedings of the 20 th National Conference on Artificial Intelligence (AAAI), pages 1,037–1,042, Pittsburgh, PA. find
  14. Chan, Yee Seng and Hwee Tou Ng. 2005b. Word Sense Disambiguation with distribution estimation. In Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI’05, pages 1,010–1,015, Edinburgh. find
  15. Chan, Yee Seng and Hwee Tou Ng. 2007. Domain adaptation with active learning for Word Sense Disambiguation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 49–56, Prague. find
  16. Chan, Yee Seng, Hwee Tou Ng, and Zhi Zhong. 2007. NUS-PT: Exploiting parallel texts for Word Sense Disambiguation in the English all-words tasks. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 253–256, Prague.
  17. Cuadros, Montse and German Rigau. 2006. Quality assessment of large scale knowledge resources. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 534–541, Sydney. find
  18. Cuadros, Montse and German Rigau. 2007. SemEval-2007 task 16: Evaluation of wide coverage knowledge resources. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 81–86, Prague. find
  19. Cuadros, Montse and German Rigau. 2008. KnowNet: Building a large net of knowledge from the Web. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 161–168, Manchester. find
  20. Curran, James R. and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the Tenth Conference of the European Chapter of the Association for Computational Linguistics - Volume 1, pages 91–98, Budapest. find
  21. Di Marco, Antonio and Roberto Navigli. 2013. Clustering and diversifying Web search results with graph-based Word Sense Induction. Computational Linguistics, 39(3):709–754.
  22. Erk, Katrin. 2007. A simple, similarity-based model for Selectional Preferences. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 216–223, Prague. find
  23. Erk, Katrin, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse Selectional Preferences. Computational Linguistics, 36(4):723–763.
  24. Escudero, Gerard, Lluı́s Màrquez, and German Rigau. 2000. An empirical study of the domain dependence of supervised Word Sense Disambiguation systems. In Proceedings of the 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora: Held in Conjunction w… find
  25. Faralli, Stefano and Roberto Navigli. 2012. A new minimally-supervised framework for Domain Word Sense Disambiguation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1,411–1,422, Jeju. find
  26. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
  27. Flati, Tiziano and Roberto Navigli. 2012. The CQC algorithm: Cycling in graphs to semantically enrich and enhance a bilingual dictionary. Journal of Artificial Intelligence Research (JAIR), 43:135–171. find
  28. Gale, William, Kenneth Church, and David Yarowsky. 1992a. Work on statistical methods for Word Sense Disambiguation. In Proceedings of the AAAI Fall Symposium on Probabilistic Approaches to Natural Language, pages 54–60, Cambridge, MA. find
  29. Gale, William A., Kenneth Church, and David Yarowsky. 1992b. A method for disambiguating word senses in a corpus. Computers and the Humanities, 26:415–439. find
  30. Gaustad, Tanja. 2001. Statistical corpus-based word sense disambiguation: Pseudowords vs. real ambiguous words. In Proceedings of the Student Research Workshop of the 39th Annual Meeting of the Association for Computational Linguistics (ACL/EACL 2001), pages 61–66, Toulouse. find
  31. Graff, David and Christopher Cieri. 2003. English gigaword, LDC2003T05. In Linguistic Data Consortium. Philadelphia, PA. find
  32. Haveliwala, Taher H. 2002. Topic-sensitive PageRank. In Proceedings of the 11 th International Conference on World Wide Web (WWW 2002), pages 517–526, Honolulu, HI. find
  33. Hovy, Eduard H., Roberto Navigli, and Simone Paolo Ponzetto. 2013. Collaboratively built semi-structured content and artificial intelligence: The story so far. Artificial Intelligence, 194:2–27. find
  34. Hughes, Thad and Daniel Ramage. 2007. Lexical semantic relatedness with random graph walks. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’07, pages 581–589, Prague. find
  35. Ide, Nancy, Collin F. Baker, Christiane Fellbaum, and Rebecca J. Passonneau. 2010. The manually annotated sub-corpus: A community resource for and by the people. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages 68–73, Uppsala. find
  36. Jurgens, David and Keith Stevens. 2011. Measuring the impact of sense similarity on Word Sense Induction. In Proceedings of the First Workshop on Unsupervised Learning in NLP, EMNLP ’11, pages 113–123, Edinburgh. find
  37. Khapra, Mitesh, Anup Kulkarni, Saurabh Sohoney, and Pushpak Bhattacharyya. 2010. All words domain adapted WSD: Finding a middle ground between supervision and unsupervision. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,532–1,541, Uppsala. find
  38. Kilgarriff, Adam and Joseph Rosenzweig. 2000. English Senseval: Report and results. In Proceedings of the 2 nd Conference on Language Resources and Evaluation (LREC), pages 1,239–1,244, Athens.
  39. Leacock, Claudia, Martin Chodorow, and George Miller. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147–166.
  40. Lee, Yoong Keok and Hwee Tou Ng. 2002. An empirical evaluation of knowledge sources and learning algorithms for Word Sense Disambiguation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’02, pages 41–48, Philadelphia, PA.
  41. Lin, Chin-Yew and Eduard Hovy. 2000. The automated acquisition of topic signatures for text summarization. In Proceedings of the 18th Conference on Computational Linguistics - Volume 1, COLING ’00, pages 495–501, Saarbrücken.
  42. Lin, Dekang. 1998. An information-theoretic definition of similarity. In Proceedings of the 15 th International Conference on Machine Learning, pages 296–304, Madison, WI. find
  43. Litkowski, Ken. 2004. Senseval-3 task: Word Sense Disambiguation of WordNet glosses. In Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 13–16, Barcelona. find
  44. Lu, Zhimao, Haifeng Wang, Jianmin Yao, Ting Liu, and Sheng Li. 2006. An equivalent pseudoword solution to Chinese Word Sense Disambiguation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics,… find
  45. Marcus, Mitchell, Grace Kim, Mary Ann Marcinkiewicz, Robert Macintyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In ARPA Human Language Technology Workshop, pages 114–119, Plainsboro, NJ.
  46. Martinez, David. 2004. Supervised Word Sense Disambiguation: Facing Current Challenges. Ph.D. Thesis. University of the Basque Country, Spain. find
  47. Martinez, David, Oier Lopez de Lacalle, and Eneko Agirre. 2008. On the use of automatically acquired examples for all-nouns Word Sense Disambiguation. Journal of Artificial Intelligence Research, 33(1):79–107. find
  48. Matuschek, Michael and Iryna Gurevych. 2013. Dijkstra-WSA: A graph-based approach to word sense alignment. Transactions of the Association for Computational Linguistics (TACL), (1):151–164. find
  49. McCarthy, Diana, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), pages 279–286, Barcelona.
  50. Mihalcea, Rada. 2002. Bootstrapping large sense tagged corpora. In Proceedings of the 3 rd International Conference on Language Resources and Evaluations (LREC), pages 1,407–1,411, Las Palmas. find
  51. Mihalcea, Rada. 2007. Using Wikipedia for automatic Word Sense Disambiguation. In Proceedings of NAACL-HLT-07, pages 196–203, Rochester, NY. find
  52. Mihalcea, Rada, Timothy Chklovski, and Adam Kilgarriff. 2004. The Senseval-3 English lexical sample task. In Proceedings of Senseval-3: The Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 25–28, Barcelona.
  53. Mihalcea, Rada and Dan Moldovan. 1999. An automatic method for generating sense tagged corpora. In Proceedings AAAI ’99, pages 461–466, Orlando, FL. find
  54. Mihalcea, Rada and Dan Moldovan. 2001. eXtended WordNet: Progress report. In Proceedings of the NAACL Workshop on WordNet and Other Lexical Resources, pages 95–100, Pittsburgh, PA. find
  55. Miller, George A., R. T. Beckwith, Christiane D. Fellbaum, D. Gross, and K. Miller. 1990. WordNet: An online lexical database. International Journal of Lexicography, 3(4):235–244. find
  56. Miller, George A., Claudia Leacock, Randee Tengi, and Ross Bunker. 1993. A semantic concordance. In Proceedings of the 3rd DARPA Workshop on Human Language Technology, pages 303–308, Plainsboro, NJ.
  57. Moro, Andrea, Roberto Navigli, Francesco Maria Tucci, and Rebecca J. Passonneau. 2014. Annotating the MASC Corpus with BabelNet. In Proceedings of LREC, pages 4,214–4,219, Reykjavic. find
  58. Moro, Andrea, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: A unified approach. Transactions of the Association for Computational Linguistics, 2:231–244.
  59. Nakov, Preslav I. and Marti A. Hearst. 2003. Category-based pseudowords. In HLT-NAACL 2003–Short Papers, pages 67–69, Edmonton. find
  60. Navigli, Roberto. 2005. Semi-automatic extension of large-scale linguistic knowledge bases. In Proceedings of FLAIRS-05, pages 548–553, Clearwater Beach, FL. find
  61. Navigli, Roberto. 2008. A structural approach to the automatic adjudication of word sense disagreements. Journal of Natural Language Engineering, 14(4):293–310. find
  62. Navigli, Roberto. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
  63. Navigli, Roberto. 2012. A quick tour of word sense disambiguation, induction and related approaches. In Proceedings of the 38th Conference on Current Trends in Theory and Practice of Computer Science (SOFSEM), pages 115–129, Spindleruv Mlyn. find
  64. Navigli, Roberto and Mirella Lapata. 2010. An experimental study on graph connectivity for unsupervised Word Sense Disambiguation. IEEE Transactions on Pattern Anaylsis and Machine Intelligence, 32(4):678–692. find
  65. Navigli, Roberto and Simone Paolo Ponzetto. 2012a. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217–250. find
  66. Navigli, Roberto and Simone Paolo Ponzetto. 2012b. Joining forces pays off: Multilingual joint Word Sense Disambiguation. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1,399–1,410, Jeju. find
  67. Navigli, Roberto and Daniele Vannella. 2013. SemEval-2013 task 11: Evaluating Word Sense Induction and Disambiguation within an end-user application. In Proceedings of the 7 th International Workshop on Semantic Evaluation (SemEval 2013), in conjunction with the Second Joint Conference on Lexical an… find
  68. Otrusina, Lubomir and Pavel Smrz. 2010. A new approach to pseudoword generation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC), pages 1,195–1,199, Valletta. find
  69. Palmer, Martha, Hoa Dang, and Christiane Fellbaum. 2007. Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2):137–163. find
  70. Pham, Thanh Phong, Hwee Tou Ng, and Wee Sun Lee. 2005. Word Sense Disambiguation with semi-supervised learning. In Proceedings of the 20th National Conference on Artificial Intelligence, AAAI’05, pages 1,093–1,098, Pittsburgh, PA. find
  71. Pilehvar, Mohammad Taher, David Jurgens, and Roberto Navigli. 2013. Align, disambiguate and walk: A unified approach for measuring semantic similarity. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1,341–1,351, Sofia. find
  72. Pilehvar, Mohammad Taher and Roberto Navigli. 2013. Paving the way to a large-scale pseudosense-annotated dataset. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013), pages 1,100–1,109, At… find
  73. Ponzetto, Simone Paolo and Roberto Navigli. 2010. Knowledge-rich Word Sense Disambiguation rivaling supervised system. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,522–1,531, Uppsala. find
  74. Pradhan, Sameer, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007a. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92, Prague.
  75. Pradhan, Sameer S., Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007b. find
  76. OntoNotes: A unified relational semantic representation. In Proceedings of the 1st International Conference on Semantic Computing (ICSC), pages 517–526, Irvine, CA. find
  77. Sanderson, Mark and C. J. Van Rijsbergen. 1999. The impact on retrieval effectiveness of skewed frequency distributions. ACM Transactions on Information Systems, 17:440–465. find
  78. Schütze, Hinrich. 1992. Dimensions of meaning. In Supercomputing ’92: Proceedings of the 1992 ACM/IEEE Conference on Supercomputing, pages 787–796, Los Alamitos, CA. find
  79. Shen, Hui, Razvan Bunescu, and Rada Mihalcea. 2013. Coarse to fine grained sense disambiguation in Wikipedia. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 22–31, Atlanta,… find
  80. Snow, Rion, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263, Honolulu, HI.
  81. Snyder, Benjamin and Martha Palmer. 2004. The English all-words task. In Proceedings of the 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text (SENSEVAL-3), pages 41–43, Barcelona. find
  82. Vannella, Daniele, David Jurgens, Daniele Scarfini, Domenico Toscani, and Roberto Navigli. 2014. Validating and extending semantic knowledge bases using video games with a purpose. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1,294–1,304, Baltimor… find
  83. Venhuizen, Noortje J., Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In Proceedings of the International Conference on Computational Semantics (IWCS), pages 397–403, Potsdam. find
  84. Wang, Xinglong and John Carroll. 2005. Word Sense Disambiguation using sense examples automatically acquired from a second language. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 547–554, Vancouver. find
  85. Yarowsky, David. 1993. One sense per collocation. In Proceedings of the 3rd DARPA Workshop on Human Language Technology, pages 266–271, Princeton, NJ.
  86. Yarowsky, David. 1995. Unsupervised Word Sense Disambiguation rivaling supervised methods. In Proceedings of the 33 rd Annual Meeting of the Association for Computational Linguistics, pages 189–196, Cambridge, MA.
  87. Zhong, Zhi and Hwee Tou Ng. 2009. Word Sense Disambiguation for all words without hard labor. In Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI), pages 1,616–1,622, Pasadena, CA. find
  88. Zhong, Zhi and Hwee Tou Ng. 2010. It makes sense: A wide-coverage Word Sense Disambiguation system for free text. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 78–83, Uppsala. find
  89. Zipf, George K. 1949. Human Behavior and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pilehvar-2014-large-scale-pseudoword-based-evaluation/