Elamur Research⌕
☰
home / papers / 2020 / IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Summary

Indonesian has a very large online user base, but NLP work on it has advanced slowly. Datasets are scattered and poorly documented, and many studies omit code or test splits, so results cannot be reproduced. The authors respond with IndoNLU, the first Indonesian NLU benchmark. It gathers twelve existing datasets covering formal and colloquial text across several domains and gives them standardized splits. The tasks fall into four groups: single-sentence classification, sentence-pair classification, single-sentence sequence labeling, and sentence-pair sequence labeling. Alongside the benchmark they assemble Indo4B, a cleaned pre-training corpus drawn from news, social media, Wikipedia, articles, subtitles and parallel data. On this corpus they train monolingual IndoBERT models and lighter ALBERT-style IndoBERT-lite models, then compare them with fastText embeddings and multilingual models such as mBERT and XLM-R. The monolingual models lead on most tasks, especially classification. Multilingual models keep an edge on tasks rich in English loanwords and foreign names. The released data, code, models and public leaderboard give Indonesian NLP a shared, reproducible reference point.

Key points

  • Benchmark scope: IndoNLU has 12 tasks, including:
    • EmoT, SmSA, CASA and HoASA (classification)
    • WReTE (entailment)
    • POSP and BaPOS (POS tagging)
    • NERGrit and NERP (NER)
    • TermA, KEPS and FacQA (span extraction)
  • Pre-training corpus: Indo4B contains roughly 4 billion words (≈23 GB). fastText trained on it beat fastText trained on the much larger CC-ID corpus (≈180 GB) on 10 of 12 tasks.
  • Model sizes: IndoBERT comes in BASE (124.5M) and LARGE (335.2M) parameter sizes. IndoBERT-lite comes in BASE (11.7M) and LARGE (17.7M).
  • Monolingual vs. multilingual: IndoBERT models outperformed multilingual baselines on 8 of 12 tasks. XLM-R remained stronger on POSP, NERGrit and FacQA.
  • Efficiency and release: IndoBERT-lite LARGE matches XLM-R BASE with about 16× fewer parameters. The leaderboard is hosted at indobenchmark.com.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

84 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.aacl-main.85

@inproceedings{wilie2020indonlu,
  title = {{IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding}},
  author = {Wilie, Bryan and Vincentio, Karissa and Winata, Genta Indra and Cahyawijaya, Samuel and Li, Xiaohong and Lim, Zhi Yuan and Soleman, Sidik and Mahendra, Rahmad and Fung, Pascale and Bahar, Syafri and Purwarianti, Ayu},
  booktitle = {Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.85},
  url = {https://doi.org/10.18653/v1/2020.aacl-main.85}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić and Ivan Vulić. 2019. Jw300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3204–3210. find our card
  2. A. N. Azhar, M. L. Khodra, and A. P. Sutiono. 2019. Multi-label aspect categorization with convolutional neural networks and extreme gradient boosting. In 2019 International Conference on Electrical Engineering and Informatics (ICEEI), pages 35–40. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. find
  4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  6. Arawinda Dinakaramani, Fam Rashel, Andry Luthfi, and Ruli Manurung. 2014. Designing an indonesian part of speech tagset and manually tagged indonesian corpus. In 2014 International Conference on Asian Language Processing, IALP 2014, Kuch-ing, Malaysia, October 20-22, 2014, pages 66–69. IEEE. find
  7. Jordhy Fernando, Masayu Leylia Khodra, and Ali Akbar Septiandri. 2019. Aspect and opinion terms extraction using double embeddings and attention mechanism for indonesian hotel reviews. find
  8. Filip Ginter, Jan Hajič, Juhani Luotolahti, Milan Straka, and Daniel Zeman. 2017. find
  9. CoNLL 2017 shared task - automatically annotated raw texts and word embeddings. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  10. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find our card
  11. Devin Hoesen and Ayu Purwarianti. 2018. Investigating bi-lstm and crf with pos tag embedding for indonesian named entity tagger. In 2018 International Conference on Asian Language Processing (IALP), pages 35–38. IEEE. find
  12. Arfinda Ilmania, Samuel Cahyawijaya, Ayu Purwarianti, et al. 2018. Aspect detection and sentiment classification using deep neural network for indonesian aspect-based sentiment analysis. In 2018 International Conference on Asian Language Processing (IALP), pages 62–67. IEEE. find
  13. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. find our card
  14. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… find our card
  15. Kemal Kurniawan and Alham Fikri Aji. 2018. Toward a standardized and more accurate indonesian part-of-speech tagging. In 2018 International Conference on Asian Language Processing (IALP), pages 303– 307. IEEE. find
  16. Andrey Kutuzov and Elizaveta Kuzmenko. 2019. To lemmatize or not to lemmatize: How word normalisation affects ELMo performance in word sense disambiguation. In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pages 22–28, Turku, Finland. Linköping University… find
  17. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. find
  18. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  19. Hang Le, Loı̈c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoı̂t Crabbé, Laurent Besacier, and Didier Schwab. 2019. Flaubert: Unsupervised language model pre-training for french. find our card
  20. Hiroki Nomoto, Kenji Okano, David Moeljadi, and Hideo Sawada. 2018. Tufs asian language parallel corpus (talpco). In Proceedings of the Twenty-fourth Annual Meeting of the Association for Natural Language Processing, pages 436–439. find
  21. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Spr… find
  22. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  23. Femphy Pisceldo, Rahmad Mahendra, Ruli Manurung, and I Wayan Arka. 2008. A two-level morphological analyser for the indonesian language. In Proceedings of the Australasian Language Technology Association Workshop 2008, pages 142–150. find
  24. Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019. Improving bi-lstm performance for indonesian sentiment analysis using paragraph vector. In 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pages 1–5. IEEE. find
  25. Pierre Lison and Jörg Tiedemann. 2016. Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation. find our card
  26. Ayu Purwarianti, Masatoshi Tsuchiya, and Seiichi Nakagawa. 2007. A machine learning approach for indonesian question answering system. In Artificial Intelligence and Applications, pages 573–578. find
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  28. Mei Silviana Saputri, Rahmad Mahendra, and Mirna Adriani. 2018. Emotion classification on indonesian twitter dataset. In 2018 International Conference on Asian Language Processing (IALP), pages 90–95. IEEE. find
  29. Miftahul Mahfuzh, Sidik Soleman, and Ayu Purwarianti. 2019. Improving joint layer rnn based keyphrase extraction by using syntactical features. In 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pages 1–6. IEEE. find
  30. Ali Akbar Septiandri and Arie Pratama Sutiono. 2019. Aspect and opinion term extraction for aspect based sentiment analysis of hotel reviews using transfer learning. find
  31. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2019. Camembert: a tasty french language model. find our card
  32. David Moeljadi. 2012. Usage of indonesian possessive verbal predicates: a statistical analysis based on questionnaire and storytelling surveys. In APLL-5 conference. SOAS, University of London. find
  33. Ken Nabila Setya and Rahmad Mahendra. 2018. Semi-supervised textual entailment on indonesian wikipedia data. In 2018 International Conference on Computational Linguistics and Intelligent Text Processing (CICLing). find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  35. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261–3275. find our card
  36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. find our card
  37. Liang Xu, Xuanwei Zhang, Lu Li, Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilie-2020-indonlu-benchmark-resources-evaluating-indonesian/