📄 The Role of Context Types and Dimensionality in Learning Word Embeddings

Summary

Skip-gram embeddings are usually trained on fixed-size word windows. This paper asks how the choice of context, together with vector size, affects results across many tasks. The authors trained all models on one shared English corpus of roughly 10B words, built from Wikipedia 2015, UMBC and Gigaword and parsed with Stanford CoreNLP. They compared three context types:

• window contexts of sizes 1, 5 and 10
• dependency contexts, trained with word2vecf
• a new weighted substitute-word variant, where language-model fillers act as weighted contexts

They evaluated these on word similarity and relatedness benchmarks and on four downstream tasks: dependency parsing, NER, coreference and sentiment analysis. They also tested combinations, either by concatenating vectors or by merging them with SVD or CCA. On intrinsic benchmarks, particular context types and larger dimensionality were consistently preferred. Downstream tasks instead needed task-specific tuning of both settings. Once adding dimensions stopped helping, concatenating embeddings from different context types often brought further gains. The practical lesson is to choose embeddings for an applied system by testing them on that task, not by relying on intrinsic benchmark scores.

Key points

• Training setup: Words seen fewer than 100 times were dropped, leaving a vocabulary of about 500K. Parses follow Universal Dependencies (Stanford CoreNLP 3.5.2), and the embeddings were released publicly.
• Substitute-based embeddings (SUB):
  • A 4-gram Kneser-Ney model built with KenLM, plus FASTSUBS, produced the substitute vectors.
  • Each vector was cut to its top-10 substitutes with normalized probabilities, and at most 20,000 vectors were generated per word type.
  • The word2vecf objective was changed so that each target–context pair counts in proportion to its weight.
• Qualitative differences: Large windows (W10) capture topical relatedness. Small windows and DEP favour functional similarity. SUB is the most functional and even tends to preserve verb tense.
• Concatenation:
  • It helps only once single embeddings are near saturation. For example, in sentiment at 50 dimensions, joining two 25-dimensional vectors did worse than one 50-dimensional vector.
  • The best combinations usually paired topical W10 with a functional type (SUB, DEP or W1).
• SVD and CCA merging: Both performed worse than single embeddings on every extrinsic task. This supports the view that different context types carry complementary information.