📄 Why is unsupervised alignment of English embeddings from different algorithms so hard?

Summary

The authors ask when unsupervised bilingual dictionary induction can work at all. To remove differences between languages, they align English with English. They train 300-dimensional embeddings with five algorithms (Hyperwords-SVD, Hyperwords-SGNS, CBOW, GloVe, FastText) on two disjoint 10% samples of a March 2018 English Wikipedia dump. They then align the spaces with MUSE, a GAN with a linear generator followed by Procrustes refinement, and compare against supervised Procrustes alignment that uses 5000 seed words.

The result is stark. Pairs built with the same algorithm align almost perfectly. Pairs built with different algorithms reach zero precision, even when trained on the same text. Supervised Procrustes still maps those failing pairs almost perfectly, so a linear map does exist. Normalization, centering and hyper-parameter changes do not help. The authors suspect that differing inductive biases create a loss surface full of local optima, which traps the discriminator. They present this as an open challenge for the stability of adversarial alignment and unsupervised MT.

Key points

• Data: Each Wikipedia sample has over 400 million tokens. The vocabularies have 463,576 and 528,556 words, with 351,858 shared.

• Evaluation and main results: Alignment is scored by Precision@1 on the 1500 test words from Bojanowski et al. (2017).

  • Same-algorithm pairs score 0.992–0.997.
  • Every cross-algorithm pair scores 0.000 in the unsupervised setting.
  • Supervised Procrustes reaches 0.967–0.999 on those same cross-algorithm pairs.
  • Precision@10 shows the same pattern.

• MUSE settings: The defaults were used. The discriminator has 2 fully connected layers of 2048 units and is trained for 5 epochs of 1,000,000 iterations, with 5 discriminator steps per iteration and batch size 32. The generator is initialized as the identity matrix.

• Embedding geometry: Average inner product with the mean vector differs strongly across algorithms:

  Algorithm	Avg. inner product with mean
  Hyperwords-SVD	0.0032
  Hyperwords-SGNS	0.0002
  GloVe	0.0002
  FastText	0.2988
  CBOW	4.2985

  However, normalizing and/or centering the embeddings still leaves cross-algorithm P@1 at 0.

• Extra analyses:

  • A learning curve for FastText (evaluated on 878 shared words) shows how much data a same-algorithm alignment needs.
  • Discriminator loss curves show two distinct behaviours. Same-algorithm runs display a genuine min-max contest. Cross-algorithm runs see the discriminator loss collapse early and stay stuck.

Interesting ideas

• I1. Pairing the unsupervised method with a supervised upper bound on the same pair separates "no solution exists" from "the optimizer cannot find it," which is a cleaner diagnosis than reporting a failure alone.
• I2. Aligning a language with itself removes linguistic variation and gives a control condition where success should be trivial, so any failure exposes the method's own weakness.
• I3. Ruling out cheap explanations one by one (hyper-parameters, normalization, centering, initialization) before proposing a deep cause keeps the final claim honest.
• I4. Watching training dynamics, here whether two adversaries keep pushing each other or one collapses early, is a fast health signal for a run that might otherwise look merely "bad."
• I5. A single geometric statistic, the average inner product with the mean vector, quickly reveals how differently two representation spaces are shaped.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When a self-induced repair rule or silver-data learner fails on a phenomenon, first check whether a supervised fit on gold examples succeeds, so search failures are not mistaken for missing signal.
• I2 (learn): Run Mova's agreement and evaluation pipelines on deliberately self-identical inputs, such as one treebank split two ways, as a negative or positive control before trusting cross-source comparisons.
• I3 (learn): In error analysis of the witness parsers, write down and test the trivial explanations for a disagreement before attributing it to a real linguistic ambiguity.
• I4 (tree): Log loss trajectories of gparse training runs and flag runs whose curves flatten abnormally early, as a cheap sign of a stuck optimizer rather than a data problem.
• I5 (world): When comparing compiled knowledge vectors or ontology embeddings from different sources (WordNet, Wikidata), compute such summary statistics first to anticipate whether they can be mapped onto each other.