Elamur Research⌕
☰
home / papers / 2020 / An Empirical Study of Pre-trained Transformers for Arabic Information Extraction

An Empirical Study of Pre-trained Transformers for Arabic Information Extraction

Paper · 2020 · Wuwei Lan, Yang Chen, Wei Xu et al.

Summary

Multilingual encoders such as mBERT and XLM-RoBERTa can be fine-tuned on English labels and then applied to Arabic. Their results on Arabic information extraction had not been studied closely, and this zero-shot setting still falls well short of supervised training. The authors build GigaBERT, a bilingual English–Arabic BERT-base model. It is pre-trained mainly on newswire from the English and Arabic Gigaword corpora, with Wikipedia and the Arabic part of Oscar added and Arabic data up-sampled. Five versions differ in tokenizer (separate SentencePiece vocabularies vs. one shared WordPiece vocabulary), casing, sequence length and data. The final version, GigaBERT-v4, continues training on synthetic code-switched text built from bilingual dictionaries.

Evaluation covers four tasks: NER, POS tagging, argument role labeling and relation extraction. The data come from ACE 2005 and the UD Treebank v1.4, tested both with Arabic supervision and with English-to-Arabic zero-shot transfer. GigaBERT beats mBERT, XLM-R base and the Arabic-only AraBERT. GigaBERT-v4 sets a new zero-shot state of the art on NER, ARL and RE. The study suggests that a model tailored to one language pair and domain can outperform huge multilingual ones.

Key points

  • Pre-training data: up to 10.4B tokens (6.1B English / 4.3B Arabic) from Gigaword 5th edition (13M news articles), Wikipedia and Oscar. The model uses a 50k vocabulary and has about 125M parameters, compared with 270M for XLM-R base.
  • Code-switching recipe for v4:
    • Up to 50% of sentences are altered, with up to 30% of tokens replaced per sentence.
    • Substitutions come first from Wikipedia titles (2M entries), then PanLex (24K), then MUSE (44K).
    • In practice 47.4% of sentences were switched, and MUSE proved the most useful dictionary.
  • Zero-shot Arabic NER F1: 51.5 for GigaBERT-v4, versus 30.8 for mBERT and 43.5 for XLM-R base. With Arabic supervision, v4 reaches 84.1 NER F1, versus 78.6 for AraBERT.
  • Shared subwords are not essential: GigaBERT-v3 shares almost no word pieces or script between the two languages, yet it still transfers strongly. This contradicts the view that shared subwords drive cross-lingual ability.
  • Embedding analysis: the authors compared 5,340 GALE parallel sentence pairs with random pairs.
    • GigaBERT keeps the two groups clearly apart in all 12 layers.
    • XLM-R scores both groups as highly similar.
    • mBERT scores both groups low.
    • Separately, continued pre-training of XLM-R on only about 1% of Gigaword (GigaXLM-R) already improved zero-shot NER, POS and RE.

Interesting ideas

  • I1. Dictionary-based code-switching with hard caps (at most half the sentences and 30% of tokens per sentence) creates cross-lingual anchor points while keeping most training text real.
  • I2. Which checkpoint looks best depends on which language's dev set is used for selection, so the top single-language checkpoint is not automatically the top transfer checkpoint.
  • I3. Comparing cosine similarity of true parallel pairs against random pairs, layer by layer, is a cheap probe for whether representations carry meaning rather than surface similarity.
  • I4. For argument roles, every non-argument entity in the same sentence as a trigger serves as a negative example, which gives hard, context-matched negatives for free.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when enlarging silver data with lexical substitutions (e.g., WordNet synonyms or VerbNet class-mates), cap the share of altered sentences so the parser still sees mostly natural text.
  • I2 (learn): pick gparse checkpoints on a dev set from the target dialect or genre (e.g., fables, word problems), not only on the source treebank's dev split.
  • I3 (learn): use random-pair baselines as a negative control when checking that encoder or graph representations separate paraphrases from unrelated sentences.
  • I4 (frames): train or check semantic-role assignment by contrasting a verb's true arguments with the other entities in its own sentence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓The Ohio State UniversityColumbus · United States82 works · 18,502 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 worksInformation extraction · 167 works

Cite

Cite as Wuwei Lan, Yang Chen, Wei Xu and Alan Ritter. 2020. An Empirical Study of Pre-trained Transformers for Arabic Information Extraction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.382

@inproceedings{lan2020empirical,
  title = {{An Empirical Study of Pre-trained Transformers for Arabic Information Extraction}},
  author = {Wuwei Lan, Yang Chen, Wei Xu and Alan Ritter},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.382},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.382}
}

References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the International Conference on Language Resources and Evaluation. ↗
  2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the Association for Computational Linguistics. Proceedings of the Association for Computational Linguistics. find
  3. Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural crf model for sentence alignment in text simplification. In Proceedings of the Association for Computational Linguistics.
  4. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual BERT: An empirical study. In Proceedings of the International Conference on Learning Representations. ↗
  5. David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building a resource for panlingual lexical translation. In Proceedings of the International Conference on Language Resources and Evaluation. find
  6. Hyun Kim, Joon-Ho Lim, Hyun-Ki Kim, and Seung-Hoon Na. 2019. QE BERT: Bilingual BERT using multi-task learning for neural quality estimation. In Proceedings of the Fourth Conference on Machine Translation. find
  7. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representation. ↗
  8. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of International Conference on Learning Representation. ↗
  9. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of Emperical Methods in Natural Language Processing.
  10. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the Association for Computational Linguistics. ↗
  11. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In Proceedings of Advances in Neural Information Processing Systems. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ↗
  13. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. 2019. hULMonA: The universal language model in Arabic. In Proceedings of the Fourth Arabic Natural Language Processing Workshop.
  14. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English Gigaword. Linguistic Data Consortium. find
  15. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  16. Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of Emperical Methods in Natural Language Processing.
  17. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford CoreNLP natural language processing toolkit. In Proceedings of the Association for Computational Linguistics. Joakim Nivre, Željko Agić, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Yevgeni Berzak, Riyaz Ahmad Bhat, Eckhard Bick, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Gülşen Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Fabricio Chalub, Çağrı Çöltekin, Miriam Connor, Elizabeth Davidson, Marie-Catherine de Marneffe, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Claudia Freitas, Katarı́na Gajdošová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Grūzı̄tis, Bruno Guillaume, Jan Hajič, Linh Hà Mỹ, Dag Haug, Barbora Hladká, Radu Ion, Elena Irimia, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşıkara, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Jessica Kenney, Natalia Kotsyba, Simon Krek, Veronika Laippala, Lucia Lam, Phuong Lê H`ông, Alessandro Lenci, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Christopher Manning, Cătălina Mărănduc, David Mareček, Héctor Martı́nez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Keiko Sophie Mori, Shunsuke Mori, Bohdan Moskalevskyi, Kadri Muischnek, Nina Mustafina, Kaili Müürisep, Lu’o’ng Nguy˜ên Thi., Huy`ên Nguy˜ên Thi.
  18. Minh, Vitaly Nikolaev, Hanna Nurmi, Petya Osenova, Robert Östling, Lilja Øvrelid, Valeria Paiva, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Slav Petrov, Jussi Piitulainen, Barbara Plank, Martin Popel, Lauma Pretkalniņa, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy, Livy Real, Laura Rituma, Rudolf Rosa, Shadi Saleh, Baiba Saulı̄te, Sebastian Schuster, Wolfgang Seeker, Mojgan Seraji, Lena Shakurova, Mo Shen, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Carolyn Spadine, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Takaaki Tanaka, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Larraitz Uria, Gertjan van Noord, Viktor Varga, Veronika Vincze, Lars Wallin, Jing Xian Wang, Jonathan North Washington, Mats Wirén, Zdeněk Žabokrtský, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. 2016. Universal Dependencies 1.4. find
  19. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of Advances in Neural Information Processing Systems. ↗
  20. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the Association for Computational Linguistics.
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems.
  22. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. ACE 2005 multilingual training corpus. Linguistic Data Consortium, Philadelphia, 57. ↗
  23. Shijie Wu and Mark Dredze. 2019. Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing. ↗
  24. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. ↗
  25. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of Advances in Neural Information Processing Systems. ↗
  26. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In Proceedings of the 7th Workshop on the Challenges in the Management of Large Corpora. find
  27. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ↗
  28. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2009. Arabic Gigaword. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lan-2020-empirical-study-pre-trained-transformers/