Elamur Research⌕
☰
home / papers / 2013 / Text: now in 2D! A framework for lexical expansion with contextual similarity

Text: now in 2D! A framework for lexical expansion with contextual similarity

Summary

The authors propose to treat text as a two-dimensional object. The usual linear axis holds syntagmatic relations, such as dependencies or word positions. A second axis is filled with paradigmatic alternatives, called lexical expansions, generated for every word with its context taken into account. The idea draws on de Saussure's structuralism and Harris's distributional hypothesis. They build a scalable framework that computes a distributional thesaurus (DT) from large corpora. A component called the holing system turns observed text structures directly into term–context-feature pairs, so no full tensor or dense vectors need to be stored. On top of the global DT, they add a contextualization method that adapts expansions to a specific, previously unseen occurrence. In a WordNet-based evaluation, their similarity measure beats earlier measures on very large corpora. In lexical substitution, contextualized expansions outperform a baseline lacking contextualization for each part of speech. The approach also helps part-of-speech tagging. Because it creates candidates itself rather than reranking a supplied list, it needs no existing lexical-semantic resources. This makes it attractive for languages with limited resources.

Key points

  • Core concept: "two-dimensional text." Dimension 1 holds syntagmatic relations; dimension 2 holds contextualized paradigmatic expansions such as similar terms.
  • A Harris-style "environment" is approximated by a context feature: a tuple of a language element plus the syntagmatic relation linking it to the target.
  • The DT follows the line of Lin (1998). Each target receives a ranked list of globally similar words, which the authors then make context-dependent.
  • Evaluations cover three areas:
    • WordNet-based judgment of DT quality, where the measure outperforms earlier measures at very large corpus scale;
    • a lexical substitution task, where it does better than a baseline without contextualization for every part of speech;
    • an application to POS tagging.
  • The method is unsupervised and knowledge-free. Unlike comparable systems, it generates expansions instead of ranking a given candidate list. (Exact scores and corpus details are in the omitted middle part of the provided text and should be checked in the original.)

Interesting ideas

  • I1. Expansion as preprocessing. Each token can be enriched with context-appropriate similar words before running WSD, text similarity, passage scoring or segmentation. This is a cheap way to bridge vocabulary mismatch in many downstream tasks, including Ukrainian ones.
  • I2. Sparse pairs instead of tensors or dense vectors. The holing system emits only observed term–feature pairs and never stores zeros. This keeps similarity computation scalable on huge corpora, and the context definition can easily be swapped (dependencies, n-gram positions, etc.).
  • I3. Separate global and contextual similarity. A global DT can be built once, and context is applied at use time to filter or reorder its entries for a specific occurrence. This two-stage design reuses one expensive resource across many contexts.
  • I4. Generate, don't just rank. Producing candidates from corpus statistics removes the dependence on WordNet-like resources. This is valuable wherever such resources are small or missing.
  • I5. Evaluate a corpus-built resource at two levels. The authors check intrinsic quality against an existing lexical resource and extrinsic usefulness on a task (lexical substitution, POS tagging). This is a sound template for validating any automatically built thesaurus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

64 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 63 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.15398/jlm.v1i1.60

@article{biemann2013text,
  title = {{Text: now in 2D! A framework for lexical expansion with contextual similarity}},
  author = {Biemann, Chris and Riedl, Martin},
  journal = {Journal of Language Modelling},
  year = {2013},
  doi = {10.15398/jlm.v1i1.60},
  url = {https://doi.org/10.15398/jlm.v1i1.60}
}

References

71 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michele Banko and Eric Brill (2001), Scaling to very very large corpora for natural language disambiguation, in Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, ACL ’01, pp. 26–33, Association for Computational Linguistics, Stroudsburg, PA, USA, http://dx.doi.org/… find
  2. Marco Baroni and Alessandro Lenci (2010), Distributional memory: A general framework for corpus-based semantics, Computational Linguistics, 36(4):673–721, ISSN 0891-2017, http://dx.doi.org/10.1162/coli_a_00016 . ↗ our card
  3. Marco Baroni and Roberto Zamparelli (2010), Nouns are vectors, adjectives are matrices: representing adjective-noun constructions in semantic space, in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pp. 1183–1193, Cambridge, Massachusetts, http://d… find
  4. Chris Biemann (2009), Unsupervised Part-of-Speech Tagging in the Large, Research on Language and Computation, 7(2–4):101–135, ISSN 1570-7075, http://dx.doi.org/10.1007/s11168-010-9067-9 . — [doi:10.1007/s11168-010-9067-9](https://doi.org/10.1007/s11168-010-9067-9) ↗
  5. Chris Biemann (2010), Co-occurrence cluster features for lexical substitutions in context, in Proceedings of the 2010 Workshop on Graph-based Methods for Natural Language Processing, TextGraphs-5, pp. 55–59, ISBN 978-1-932432-77-0, http://dl.acm.org/citation.cfm?id=1870490.1870499 . ↗
  6. Chris Biemann and Eugenie Giesbrecht (2011), Distributional Semantics and Compositionality 2011: Shared Task Description and Results, in Proceedings of the Workshop on Distributional Semantics and Compositionality, pp. 21–28, Association for Computational Linguistics, Portland, Oregon, USA, http://w… find
  7. Chris Biemann, Uwe Quasthoff, Gerhard Heyer, and Florian Holz (2008), ASV Toolbox: a Modular Collection of Language Exploration Tools, in Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008, Marrakech, Morocco, http://www.lrec-conf.org/proceedings/lrec2008/su… ↗
  8. Chris Biemann, Stefanie Roos, and Karsten Weihe (2012), Quantifying Semantics Using Complex Network Analysis, in Proceedings of the 24th International Conference on Computational Linguistics (COLING), Mumbai, India, http://aclweb.org/anthology/C/C12/C12-1017.pdf . ↗
  9. David M. Blei, Andrew Y. Ng, and Michael I. Jordan (2003), Latent Dirichlet allocation, Journal of Machine Learning Research, 3:993–1022, ISSN 1532-4435, http://dl.acm.org/citation.cfm?id=944919.944937 . ↗
  10. Stefan Bordag (2008), A comparison of co-occurrence and similarity measures as simulations of context, in CICLing’08 Proceedings of the 9th International Conference on Computational Linguistics and Intelligent Text Processing, pp. 52–63, Haifa, Israel, http://dl.acm.org/citation.cfm?id=1787578.17875… ↗
  11. Jordan Boyd-Graber and David M. Blei (2008), Syntactic Topic Models, in Neural Information Processing Systems, Vancouver, British Columbia, http://www.cs.princeton.edu/~blei/papers/Boyd-GraberBlei2009.pdf . ↗
  12. Kenneth Ward Church and Patrick Hanks (1990), Word association norms, mutual information, and lexicography, Computational Linguistics, 16(1):22–29, ISSN 0891-2017, http://dl.acm.org/citation.cfm?id=89086.89095 . ↗ our card
  13. Michael Collins (2002), Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms, in Proceedings of the ACL-02 conference on Empirical methods in natural language processing – Volume 10, EMNLP ’02, pp. 1–8, Association for Computational Linguistics,… find our card
  14. James R. Curran (2002), Ensemble methods for automatic thesaurus extraction, in Proceedings of the ACL-02 conference on Empirical methods in natural language processing – Volume 10, EMNLP ’02, pp. 222–229, http://dx.doi.org/10.3115/1118693.1118722 . ↗
  15. James R. Curran (2004), From Distributional to Semantic Similarity, University of Edinburgh, http://books.google.de/books?id=2iDbSAAACAAJ . ↗
  16. Ferdinand de Saussure (1916), Cours de linguistique générale, Payot, Paris, http://www.bibsonomy.org/bibtex/ 2e68b895a274b9569189c5ae98db84603/jntr . ↗
  17. Ferdinand de Saussure (1959), Course in general linguistics, Language (Philosophical Library), Philosophical Library, http://books.google.de/books?id=FSpZAAAAMAAJ . ↗
  18. Jeffrey Dean and Sanjay Ghemawat (2004), MapReduce: Simplified Data Processing on Large Clusters, in Proceedings of Operating Systems, Desing & Implementation (OSDI) ’04, pp. 137–150, San Francisco, CA, USA, http://doi.acm.org/10.1145/1327452.1327492 . — [doi:10.1145/1327452.1327492](https://doi.org/10.1145/1327452.1327492) ↗
  19. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman (1990), Indexing by latent semantic analysis, Journal of the American Society for Information Science, 41(6):391–407, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.108.8490 . ↗
  20. Inderjit S. Dhillon (2001), Co-clustering documents and words using bipartite spectral graph partitioning, in Proceedings of the seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’01, pp. 269–274, ACM, New York, NY, USA, ISBN 1-58113-391-X, http://doi.acm.org/10… ↗
  21. Ted Dunning (1993), Accurate methods for the statistics of surprise and coincidence, Computational Linguistics, 19(1):61–74, ISSN 0891-2017, http://dl.acm.org/citation.cfm?id=972450.972454 . ↗ our card
  22. Katrin Erk and Sebastian Padó (2008), A structured vector space model for word meaning in context, in Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pp. 897–906, Honolulu, Hawaii, http://dl.acm.org/citation.cfm?id=1613715.1613831 . ↗
  23. Stefan Evert (2005), The Statistics of Word Cooccurrences: Word Pairs and Collocations., Ph.D. thesis, Institut für Maschinelle Sprachverarbeitung, University of Stuttgart, http://elib.uni-stuttgart.de/opus/volltexte/2005/2371/ . ↗
  24. Eugenie Giesbrecht (2009), In Search of Semantic Compositionality in Vector Spaces, in Proceedings of the 17th International Conference on Conceptual Structures: Conceptual Structures: Leveraging Semantic Technologies, ICCS ’09, pp. 173–184, Springer-Verlag, Berlin, Heidelberg, ISBN 978-3-642-03078-… — [doi:10.1007/978-3-642-03079-6_14](https://doi.org/10.1007/978-3-642-03079-6_14) find
  25. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith (2011), Part-of-speech tagging for Twitter: annotation, features, and experiments, in Proceedings of the 49th Annual Meeting of the Asso… find
  26. Gene H. Golub and William M. Kahan (1965), Calculating the singular values and pseudo-inverse of a matrix, Journal of the Society for Industrial and Applied Mathematics: Series B: Numerical Analysis, 2:205–224, http://www.citeulike.org/user/rabio/article/2342309 . ↗
  27. James Gorman and James R. Curran (2006), Scaling Distributional Similarity to Large Corpora, in Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pp. 361–368, Association for Computational Linguisti… find
  28. Amit Goyal, Hal Daumé III, and Graham Cormode (2012), Sketch Algorithms for Estimating Point Queries in NLP, in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1093–1103, Association for Computational Lingu… find
  29. Emiliano Guevara (2011), Computing semantic compositionality in distributional semantics, in Proceedings of the Ninth International Conference on Computational Semantics, IWCS ’11, pp. 135–144, Association for Computational Linguistics, Stroudsburg, PA, USA, http://dl.acm.org/citation.cfm?id=2002669… ↗
  30. Zellig S. Harris (1951), Methods in Structural Linguistics, University of Chicago Press, Chicago, http://archive.org/details/structurallingui00harr . ↗
  31. W. Keith Hastings (1970), Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57(1):97–109, ISSN 1464-3510, doi:10.1093/biomet/57.1.97, http://dx.doi.org/10.1093/biomet/57.1.97 . — [doi:10.1093/biomet/57.1.97](https://doi.org/10.1093/biomet/57.1.97) ↗
  32. Marti A. Hearst (1992), Automatic acquisition of hyponyms from large text corpora, in Proceedings of the 14th conference on Computational linguistics – Volume 2, COLING ’92, pp. 539–545, http://dx.doi.org/10.3115/992133.992154 . ↗
  33. Enrique Henestroza Anguiano and Pascal Denis (2011), FreDist: Automatic construction of distributional thesauri for French, in TALN – 18ème conférence sur le traitement automatique des langues naturelles, pp. 119–124, Montpellier, France, France, http://hal.inria.fr/hal-00602004 . ↗
  34. Thomas Hofmann (1999), Probabilistic latent semantic indexing, in Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’99, pp. 50–57, ACM, New York, NY, USA, ISBN 1-58113-096-1, http://doi.acm.org/10.1145/312624.312649 . — [doi:10.1145/312624.312649](https://doi.org/10.1145/312624.312649) ↗
  35. Adam Kilgarriff, Pavel Rychly, Pavel Smrz, and David Tugwell (2004), The Sketch Engine, in Proceedings of EURALEX, http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.180.7984 . ↗
  36. Walter Kintsch (2001), Predication, Cognitive Science, 25(2):173–202, ISSN 1551-6709, http://dx.doi.org/10.1207/s15516709cog2502_1 . — [doi:10.1207/s15516709cog2502_1](https://doi.org/10.1207/s15516709cog2502_1) ↗
  37. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira (2001), Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, in Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pp. 282–289, Morgan Kaufmann Publishers Inc., San… find
  38. Lillian Lee (1999), Measures of distributional similarity, in Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, ACL ’99, pp. 25–32, College Park, Maryland, ISBN 1-55860-609-3, http://dx.doi.org/10.3115/1034678.1034693 . ↗
  39. Michael Lesk (1986), Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone, in Proceedings of the 5th Annual International Conference on Systems Documentation, SIGDOC ’86, pp. 24–26, ACM, New York, NY, USA, ISBN 0-89791-224-1, http://doi.a… ↗
  40. Dekang Lin (1998), Automatic retrieval and clustering of similar words, in Proceedings of the 17th International Conference on Computational Linguistics – Volume 2, COLING ’98, pp. 768–774, http://dx.doi.org/10.3115/980432.980696 . ↗
  41. Dekang Lin, Shaojun Zhao, Lijuan Qin, and Ming Zhou (2003), Identifying synonyms among distributionally similar words, in Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI’03, pp. 1492–1493, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, http://dl.acm… ↗
  42. Jimmy Lin and Chris Dyer (2010), Data-Intensive Text Processing with MapReduce, Morgan & Claypool Publishers, San Rafael, CA, http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.169.6896 . ↗
  43. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini (1993), Building a large annotated corpus of English: the Penn Treebank, Computational Linguistics, 19(2):313–330, ISSN 0891-2017, http://dl.acm.org/citation.cfm?id=972470.972475 . ↗ our card
  44. Marie-Catherine De Marneffe, Bill Maccartney, and Christopher D. Manning (2006), Generating typed dependency parses from phrase structure parses, in Proceedings of the International Conference on Language Resources and Evaluation, LREC 2006, Genova, Italy, http://citeseer.ist.psu.edu/viewdoc/summary… ↗ our card
  45. Diana McCarthy and Roberto Navigli (2009), The English lexical substitution task., Language Resources and Evaluation, 43(2):139–159, http://dblp.uni-trier.de/db/journals/lre/lre43.html\#McCarthyN09 . ↗
  46. George A. Miller and Walter G. Charles (1991), Contextual correlates of semantic similarity, Language and Cognitive Processes, 6(1):1–28, http://dx.doi.org/10.1080/01690969108406936 . — [doi:10.1080/01690969108406936](https://doi.org/10.1080/01690969108406936) ↗
  47. Tristan Miller, Chris Biemann, Torsten Zesch, and Iryna Gurevych (2012), Using Distributional Similarity for Lexical Expansion in Knowledge-based Word Sense Disambiguation, in Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), pp. 1781–1796, Mumbai, India, h… find
  48. Jeff Mitchell and Mirella Lapata (2008), Vector-based Models of Semantic Composition, in Proceedings of ACL-08: HLT, pp. 236–244, Columbus, Ohio, www.aclweb.org/anthology/P08-1028.pdf . find
  49. Sebastian Padó and Mirella Lapata (2007), Dependency-based construction of semantic space models, Computational Linguistics, 33(2):161–199, http://citeseer.uark.edu: 8080/citeseerx/viewdoc/summary?doi=10.1.1.86.2026 . ↗ our card
  50. Sebastian Padó and Yves Peirsman, editors (2011), Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, Association for Computational Linguistics, Edinburgh, UK, http://www.aclweb.org/anthology/W11-25 . ↗
  51. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda (2011), English Gigaword Fifth Edition, Linguistic Data Consortium, Philadelphia, http://www.ldc.upenn.edu/Catalog/catalogEntry.jsp?catalogId= LDC2011T07 . ↗
  52. Ted Pedersen, Siddharth Patwardhan, and Jason Michelizzi (2004), WordNet::Similarity: measuring the relatedness of concepts, in Demonstration Papers at HLT-NAACL 2004, HLT-NAACL – Demonstrations ’04, pp. 38–41, http://dl.acm.org/citation.cfm?id=1614025.1614037 . ↗
  53. Fernando Pereira, Naftali Tishby, and Lillian Lee (1993), Distributional clustering of English words, in Proceedings of the 31st annual meeting on Association for Computational Linguistics, ACL ’93, pp. 183–190, Association for Computational Linguistics, Stroudsburg, PA, USA, http://dx.doi.org/10.31… find
  54. Reinhard Rapp (2003), Word sense discovery based on sense descriptor dissimilarity, in Proceedings of the Ninth Machine Translation Summit, pp. 315–322, http://www.citeulike.org/user/briordan/article/2911465 . ↗
  55. Matthias Richter, Uwe Quasthoff, Erla Hallsteinsdóttir, and Chris Biemann (2006), Exploiting the Leipzig Corpora Collection, in Proceesings of the IS-LTC 2006, Ljubljana, Slovenia, http://nl.ijs.si/is-ltc06/proc/13_Richter.pdf . ↗
  56. Herbert Rubenstein and John B. Goodenough (1965), Contextual correlates of synonymy, Communications of the ACM, 8(10):627–633, ISSN 0001-0782, http://doi.acm.org/10.1145/365628.365657 . — [doi:10.1145/365628.365657](https://doi.org/10.1145/365628.365657) ↗
  57. Gerda Ruge (1992), Experiments on linguistically-based term associations, Information Processing & Management, 28(3):317 – 332, ISSN 0306-4573, http://www.sciencedirect.com/science/article/pii/030645739290078E . ↗
  58. Magnus Sahlgren (2006), The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces., Ph.D. thesis, Stockholm University, http://soda.swedish-ict.se/437/ . ↗
  59. Helmut Schmid (1995), Improvements in Part-of-Speech Tagging with an Application to German, in Proceedings of the ACL SIGDAT-Workshop, pp. 47–50, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.52.2255 . ↗
  60. Hinrich Schütze (1993), Word Space, in Advances in Neural Information Processing Systems 5, pp. 895–902, Morgan Kaufmann, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.8856 . ↗
  61. Hinrich Schütze (1998), Automatic word sense discrimination, Computational Linguistics, 24(1):97–123, ISSN 0891-2017, http://dl.acm.org/citation.cfm?id=972719.972724 . ↗ our card
  62. Anders Søgaard (2011), Semisupervised condensed nearest neighbor for part-of-speech tagging, in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: short papers – Volume 2, HLT ’11, pp. 48–52, Portland, Oregon, ISBN 978-1-932432-88-6,… find
  63. György Szarvas, Chris Biemann, and Iryna Gurevych (2013), Supervised All-Words Lexical Substitution using Delexicalized Features, in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-2013), Atlanta, G… find
  64. Ming Tan, Wenli Zhou, Lei Zheng, and Shaojun Wang (2012), A scalable distributed syntactic, semantic, and lexical language model, Computational Linguistics, 38(3):631–671, ISSN 0891-2017, http://dx.doi.org/10.1162/COLI_a_00107 . ↗
  65. Stefan Thater, Hagen Fürstenau, and Manfred Pinkal (2011), Word Meaning in Context: A Simple and Effective Vector Model, in Proceedings of 5th International Joint Conference on Natural Language Processing, pp. 1134–1143, Asian Federation of Natural Language Processing, Chiang Mai, Thailand, http://w… find
  66. Peter D. Turney and Michael L. Littman (2005), Corpus-based Learning of Analogies and Semantic Relations, Machine Learning, 60(1-3):251–278, ISSN 0885-6125, http://dx.doi.org/10.1007/s10994-005-0913-1 . — [doi:10.1007/s10994-005-0913-1](https://doi.org/10.1007/s10994-005-0913-1) ↗
  67. Peter D. Turney and Patrick Pantel (2010), From frequency to meaning: vector space models of semantics, Journal of Artificial Intelligence Research, 37(1):141–188, ISSN 1076-9757, http://dl.acm.org/citation.cfm?id=1861751.1861756 . ↗
  68. Andrew J. Viterbi (1967), Error bounds for convolutional codes and an asymptotically optimum decoding algorithm, IEEE Transactions on Information Theory, 13(2):260–269, ISSN 0018-9448, doi:10.1109/TIT.1967.1054010, http://dx.doi.org/10.1109/TIT.1967.1054010 . — [doi:10.1109/tit.1967.1054010](https://doi.org/10.1109/tit.1967.1054010) ↗
  69. Julie Weeds (2003), Measures and Applications of Lexical Distributional Similarity, Ph.D. thesis, Department of Informatics, University of Sussex, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.538 . ↗
  70. Dominic Widdows and Beate Dorow (2002), A graph model for unsupervised lexical acquisition, in Proceedings of the 19th International Conference on Computational Linguistics – Volume 1, COLING ’02, pp. 1–7, http://dx.doi.org/10.3115/1072228.1072342 . ↗
  71. This work is licensed under the Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/ ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/biemann-2013-text-now-2d-framework-lexical/