Elamur Research⌕
☰
home / papers / 2018 / Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings

Paper · 2018 · Yan Song, Shuming Shi, Jing Li et al.

Summary

The standard skip-gram (SG) model learns word vectors from co-occurrence alone and does not register whether a neighbour appears before or after the target word. Earlier order-aware extensions, such as the structured skip-gram (SSG), keep a separate output vector for each relative position. This inflates memory and training time and makes them vulnerable to sparse data. Song et al. (Tencent AI Lab) propose directional skip-gram (DSG). It gives every word one extra "direction" vector, which is trained with a sigmoid objective to tell whether a context word lies to the left or to the right. This direction score is added to the usual SG prediction. The extra cost is one operation per word, so training speed remains close to SG.

Embeddings were trained on an English Wikipedia dump of roughly 2 billion tokens. They were evaluated on word similarity and on POS tagging, the latter with a BiLSTM-CRF. DSG outperformed SG, SSG and a simplified SSG (SSSG) in both settings, and it held up especially well on a tiny training corpus. The result shows that a cheap, binary notion of word order can improve embeddings without extra resources.

Key points

  • Cost: DSG and SSSG both need 3|V|d parameters, compared with 2|V|d for SG and (2c+1)|V|d for SSG. Operations scale as 2cC(n+2)o for DSG versus 4c²C(n+1)o for SSG, so DSG's speed gap to SG narrows as the context window widens.
  • Training setup: all models shared the same hyperparameters, namely 200 dimensions, window 5, 5 negative samples, 5 iterations and a starting learning rate of 0.025.
  • Word similarity, full Wiki corpus: DSG was the best of the SG-family models, with Spearman ρ×100 of 73.76 on MEN-3k, 36.10 on SimLex-999 and 72.60 on WS-353. CWin scored higher only on MEN-3k (74.28).
  • Word similarity, 0.1% Wiki sample: this sample simulated a cold-start setting, and DSG was best on all three datasets (63.18, 27.51 and 66.71). SSG and CWin degraded markedly on the same sample.
  • POS tagging, test accuracy: on WSJ (Penn Treebank) DSG reached 97.37, ahead of SSG at 97.31 and SG at 97.12. On the ARK tweets it reached 90.43, ahead of SSG at 90.15 and SG at 88.77; the ARK embeddings came from a 100-million-tweet TWT corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Yan Song, Shuming Shi, Jing Li and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-2028

@inproceedings{song2018directional,
  title = {{Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings}},
  author = {Yan Song, Shuming Shi, Jing Li and Haisong Zhang},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-2028},
  url = {https://doi.org/10.18653/v1/n18-2028}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Douwe Kiela, Felix Hill, and Stephen Clark. 2015. Specializing word embeddings for similarity or relatedness. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2044–2048. find
  2. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring Continuous Word Representations for Dependency Parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Baltimore, Maryland, pages 809–815.
  3. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition.
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. find
  5. A Neural Probabilistic Lan-Yan Song and Chia-Jung Lee. 2017. Learning user embeddings from emails. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain, pages 733–738. find
  6. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California, pages 260–270. find
  7. Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015a. Two/Too Simple Adaptations of Word2Vec for Syntax Problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computationa…
  8. Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learning word representations with regularization from prior knowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). Vancouver, Canada, pages 143–152. find
  9. Peter D. Turney and Patrick Pantel. 2010. From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research 37(1):141–188.
  10. Wang Ling, Yulia Tsvetkov, Silvio Amir, Ramon Fermandez, Chris Dyer, Alan W Black, Isabel Trancoso, and Chu-Cheng Lin. 2015b. Not All Contexts Are Created Equal: Better Word Representations with Variable Attention. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Proces… find
  11. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv pre-print abs/1502.05698.
  12. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Baltimore, Maryland, pages 545–550.
  13. Mitch Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics 19(2):313–330.
  14. Jerrold H Zar. 1998. Spearman rank correlation. Encyclopedia of Biostatistics . find
  15. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. arXiv preprint abs/1301.3781.
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, page…
  17. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013c. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Li…
  18. Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2016. Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin, Germany, pa… find
  19. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A. Smith. 2013. Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguist…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2018-directional-skip-gram-explicitly-distinguishing/