📄 Improving Generalization in Coreference Resolution via Adversarial Training

Summary

Neural coreference resolvers trained on CoNLL-2012 may lean on memorized entity names rather than truly generalizing. The authors test this by building a "No Leakage" version of the CoNLL test set. In it, every PER and GPE name that heads a gold mention is swapped for a name never seen in training, while gender and location type are kept so the coreference structure stays intact. On this modified data, the state-of-the-art (Lee et al., 2018) model loses more than 1 CoNLL F1.

To make the model more robust, they retrain it with the fast-gradient-sign-method (FGSM) of adversarial training, used alongside dropout. Unlike the earlier work of Miyato et al., the perturbation targets the intermediate span representations rather than the input embeddings. The retrained model does better on the original CoNLL test set, on the No Leakage variant, and on the GAP pronoun dataset, reaching new state-of-the-art results on all three. The study shows that train/test name overlap inflates benchmark scores and that a cheap regularizer can partly close this gap.

Key points

• Name leakage: About 34% of the PER and GPE names heading test-set mentions also head mentions in the CoNLL training set.
• How replacement names were chosen:
  • Person names come from the 1990 U.S. Census last-name list and a first-name gender gazetteer mined from Wikipedia.
  • Place names come from GeoNames (over 11M locations) and are matched by category, with countries excluded.
  • Mention heads are found with the spaCy dependency parser.
• CoNLL F1 (original / No Leakage):
  • Lee et al. (2018): 72.96 / 71.86
  • With adversarial training: 73.23 / 72.36
  • The authors report these differences as significant only at the loose threshold of p < 0.20.
• GAP F1 (male / female / overall):
  • Lee et al. (2017): 68.7 / 60.0 / 64.5
  • Lee et al. (2018): 75.8 / 70.6 / 73.3
  • With adversarial training: 77.3 / 72.1 / 74.7
  • The female-to-male F1 ratio stays at roughly 0.93, compared with 0.87 for the 2017 model.
• Training setup:
  • The total loss mixes the original and adversarial losses with α = 0.6, and the perturbation size is ε = 1.
  • Training ran for 355,000 iterations with the original hyperparameters.
  • Perturbing character embeddings instead of span representations worked less well.