Elamur Research
☰
home / papers / 2019 / PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.11828

Summary

Adversarial paraphrase benchmarks have mostly been built only for English. PAWS, for example, contains sentence pairs that share nearly all their words but differ in meaning because of word order. Existing multilingual paraphrase corpora such as Multi30k and Opusparcus lack these hard cases. PAWS-X fills this gap by extending the Wikipedia part of PAWS to French, Spanish, German, Chinese, Japanese and Korean. Professional native-speaker translators translated sampled development and test pairs, and a commercial NMT service translated the English training set. Each translated pair keeps the original PAWS label.

The authors evaluate three baselines of rising complexity: a bag-of-words cosine model, ESIM, and multilingual BERT. They test Translate Train, Translate Test, Zero Shot and Merged regimes. Multilingual BERT fine-tuned on English plus machine-translated data in all languages performs best. Even so, accuracy remains well below the English level, especially for Chinese, Japanese and Korean. The benchmark therefore exposes weak sensitivity to structure and context in multilingual models and leaves clear headroom for cross-lingual research.

Key points

  • Dataset size: PAWS-X has 23,659 human-translated evaluation pairs, split into 11,815 dev and 11,844 test pairs. One sentence in the paper gives 23,459, but the per-split counts add up to 23,659.
    • For each language, 4,000 PAWS dev pairs were sampled and split 2,000/2,000.
    • Fewer than 2% of pairs could not be translated and were dropped.
    • The 49,401 English training pairs were machine-translated.
    • The data is available on GitHub at google-research-datasets/paws.
  • Label balance: Positive pairs make up 44.0% of dev and 45.4% of test, which is close to the original PAWS distribution.
  • Best model: BERT Merged reaches 83.1–90.8 accuracy on the non-English languages, with an average of 87.2 accuracy and 90.2 AUC-PR.
    • This is about 23 points above the best ESIM setup.
    • It is about 32 points above BOW.
    • It is 8.6 points above Zero Shot (78.6).
  • Language gap: German, French and Spanish consistently score higher than Chinese, Japanese and Korean, and the gap is largest under Zero Shot. The authors attribute this to weaker MT quality for CJK and greater typological distance from English.
  • Error analysis (BERT Merged): 61.7% of test pairs are labeled correctly in all 7 languages. Only 32 pairs fail in every language, and these are often ambiguous or have noisy original labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Cited by

267 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 250 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.11828 · arXiv 1908.11828

@misc{yang2019paws,
  title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
  author = {Yang, Yinfei and Zhang, Yuan and Tar, Chris and Baldridge, Jason},
  year = {2019},
  doi = {10.48550/arxiv.1908.11828},
  eprint = {1908.11828},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.11828}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics. find our card
  2. Hongge Chen, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. 2018. Attacking visual language grounding with adversarial examples: A case study on neural image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2587–2597, Melbourne, Australia. Association for Computational Linguistics. find
  3. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics. find
  4. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. find
  5. Mathias Creutz. 2018. Open subtitles paraphrase corpus for six languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA). find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  7. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74, Berlin, Germany. Association for Computational Linguistics. find our card
  8. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. find our card
  9. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  10. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  11. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856–865, Melbourne, Australia. Association for Computational Linguistics. find
  12. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  13. Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations. find