Elamur Research⌕
☰
home / papers / 2023 / Frank at NADI 2023 Shared Task: Trio-Based Ensemble Approach for Arabic Dialect Identification

Frank at NADI 2023 Shared Task: Trio-Based Ensemble Approach for Arabic Dialect Identification

Paper · 2023 · Dilshod Azizov, Jiyong Li and Shangsong Liang

Summary

This system paper describes the team "Frank" entry to Subtask 1 of the NADI 2023 shared task (ArabicNLP 2023). The task is to label an Arabic tweet with its country-level dialect, chosen from 18 countries. The authors trained on the new TWT-2023 data together with the organizer-supplied NADI-2021-TWT set, roughly 28K tweets in total with a fairly even label distribution. They fine-tuned three classifiers: MARBERT, and two MARBERTv2 runs (A and B) that differ only in weight decay. Each model was stopped at the epoch where it performed best. The three outputs were then combined by hard majority voting. When all three models disagree, the prediction of MARBERTv2 (B), the strongest single model, is used. On their internal evaluation split the ensemble beat every individual model. On the official test set it scored 84.76 F1 and ranked 5th of 16 teams. The work shows that a cheap ensemble of near-identical models can give a solid, competitive improvement.

Key points

  • Task: country-level dialect identification over 18 classes. Data: TWT-2023 plus NADI-2021-TWT (about 28K tweets). Other permitted resources included NADI-2020-TWT and MADAR-2018.
  • Ensemble members and settings:
    • MARBERT: 1 epoch, learning rate 5e-5, weight decay 0.001.
    • MARBERTv2 (A): 2 epochs, weight decay 0.0.
    • MARBERTv2 (B): 2 epochs, weight decay 0.001.
  • Development-side F1 scores: MARBERT 82.40, v2 (A) 84.39, v2 (B) 84.44, majority voting 85.90. The official dev set served as the test set, and 10% of the training data was held out for development.
  • Official test result: F1 84.76, accuracy 84.75, precision 84.95, recall 84.75. This placed 5th of 16 teams. Training ran on 2 NVIDIA Tesla T4 GPUs.
  • Hard voting was chosen deliberately over soft voting to sidestep calibration and overconfidence problems. Soft voting and calibration are listed as future work.

Interesting ideas

  • I1. Use hard voting with a fixed fallback: majority wins, and on a three-way split the single most reliable member decides, so every case gets a deterministic and explainable answer.
  • I2. Get ensemble diversity cheaply by training the same architecture with a small change, such as only the weight decay, instead of collecting different model families.
  • I3. Vote on discrete labels instead of averaging scores, which avoids combining confidences that are not calibrated against each other.
  • I4. Stop each ensemble member at its own best epoch rather than using one shared training schedule.
  • I5. Keep the official dev set as an untouched test proxy and carve a new dev split from training data; note that the dev gain (85.90) shrank on the real test (84.76).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When building silver data from several witness parsers, a total disagreement could default to the strongest parser while being logged as low-confidence, rather than being silently dropped or averaged.
  • I2 (tree): Extra gparse runs that differ only in regularization or seed could act as additional low-cost witnesses beside the perceptron parser.
  • I3 (tree): Mova's perceptron and transformer parsers produce scores on incomparable scales, so voting on arcs and labels is a cleaner way to merge them and easier to explain.
  • I4 (learn): Each witness or student model in the training pipeline could be checkpointed separately, so no member is overfit just to match the others' schedule.
  • I5 (learn): Evaluations should hold out a never-tuned split and report how much of any ensemble gain survives on it, as an honesty check next to the negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Dilshod Azizov, Jiyong Li and Shangsong Liang. 2023. Frank at NADI 2023 Shared Task: Trio-Based Ensemble Approach for Arabic Dialect Identification. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.67

@inproceedings{azizov2023frank,
  title = {{Frank at NADI 2023 Shared Task: Trio-Based Ensemble Approach for Arabic Dialect Identification}},
  author = {Dilshod Azizov, Jiyong Li and Shangsong Liang},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.67},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.67}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020a. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020b. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  6. Abdulkareem Alsudais, Wafa Alotaibi, and Faye Alo-mary. 2022. Similarities between arabic dialects: Investigating geographical proximity. Information Processing & Management, 59(1):102770. ↗
  7. Maha J Althobaiti. 2020. Automatic arabic dialect identification systems for written texts: A survey. arXiv preprint arXiv:2009.12622. ↗
  8. Maha J Althobaiti. 2022. Creation of annotated country-level dialectal arabic resources: An unsupervised approach. Natural Language Engineering, 28(5):607– 648. ↗
  9. Dilshod Azizov, S Liang, and P Nakov. 2023. Frank at checkthat! 2023: Detecting the political bias of news articles and news media. Working Notes of CLEF. find
  10. Alberto Barrón-Cedeño, Firoj Alam, Tommaso Caselli, Giovanni Da San Martino, Tamer Elsayed, Andrea Galassi, Fatima Haouari, Federico Ruggeri, Julia Maria Struß, Rabindra Nath Nandi, et al. 2023a. The clef-2023 checkthat! lab: Checkworthiness, subjectivity, political bias, factuality, and authority. In European Conference on Information Retrieval, pages 506–517. Springer. find
  11. Alberto Barrón-Cedeño, Firoj Alam, Andrea Galassi, Giovanni Da San Martino, Preslav Nakov, Tamer Elsayed, Dilshod Azizov, Tommaso Caselli, Gullal S Cheema, Fatima Haouari, et al. 2023b. Overview of the clef–2023 checkthat! lab on checkworthiness, subjectivity, political bias, factuality, and authority of news articles and their source. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 251–275. Springer. find
  12. Heba Elfardy, Mohamed Al-Badrashiny, and Mona Diab. 2014. Aida: Identifying code switching in informal arabic text. In Proceedings of The First Workshop on Computational Approaches to Code Switching, pages 94–101.
  13. Mohamed AE Gadalla and Waguih H ElMaraghy. 1997. Improving the accuracy of machined parametric surfaces using cutting force synthesis and surface offset techniques. In ASME International Mechanical Engineering Congress and Exposition, volume 26782, pages 181–187. American Society of Mechanical Engineers. find
  14. Clive Holes. 2004. Modern Arabic: Structures, functions, and varieties. Georgetown University Press. find
  15. Hamdy Mubarak and Kareem Darwish. 2014. Using twitter to collect a multi-dialectal corpus of arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 1–7.
  16. Omer Sagi and Lior Rokach. 2018. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249. ↗
  17. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of arabic. In LREC, pages 1240–1245.
  18. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  19. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41.
  20. Matteo Zampieri, Andrej Ceglar, Frank Dentener, and Andrea Toreti. 2017. Wheat yield loss attributable to heat waves, drought and water excess at the global, national and subnational scales. Environmental Research Letters, 12(6):064008. ↗
  21. Yadong Zhu, Xiliang Wang, Qing Li, Tianjun Yao, and Shangsong Liang. 2021. Botspot++: A hierarchical deep ensemble model for bots install fraud detection in mobile advertising. ACM Transactions on Information Systems (TOIS), 40(3):1–28. ↗
  22. Kristen Brustad. 2000. The syntax of spoken Arabic: A comparative study of Moroccan, Egyptian, Syrian, and Kuwaiti dialects. Georgetown University Press. find
  23. Giovanni Da San Martino, Firoj Alam, Maram Hasanain, Rabindra Nath Nandi, Dilshod Azizov, and Preslav Nakov. 2023. Overview of the clef-2023 checkthat! lab task 3 on political bias of news articles and news media. Working Notes of CLEF. find
  24. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. Colaba: Arabic dialect annotation and processing. In Lrec workshop on semitic language processing, pages 66–74. Citeseer. find
  25. Thomas G Dietterich et al. 2002. Ensemble learning. The handbook of brain theory and neural networks, 2(1):110–125. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/azizov-2023-frank-nadi-2023-shared-task/