Elamur Research⌕
☰
home / papers / 2026 / MaskedVerbalizer: Automatic Verbalizer Construction for Few-Shot Text Classification in Low-Resource Right-to-Left Languages

MaskedVerbalizer: Automatic Verbalizer Construction for Few-Shot Text Classification in Low-Resource Right-to-Left Languages

Summary

The paper addresses few-shot text classification for low-resource right-to-left languages (Arabic, Urdu, Sindhi, Pashto), where labelled data is scarce and rich morphology makes hand-built verbalizers unreliable. The authors propose MASKEDVERBALIZER, a self-supervised way to derive label words from the model's own predictions. The procedure has four steps:

  1. A PLM (mBERT or XLM-RoBERTa) is prompt-tuned on 4 examples per class, using a translated cloze template and two seed label words per class. The seed words are chosen manually or suggested by GPT from the 4-shot samples.
  2. The tuned model is then run on a held-out 50% of the data.
  3. For correctly classified items only, the tokens predicted at the [MASK] slot are counted for each class.
  4. The two or three most frequent tokens become the final verbalizer, with extra heuristics for label noise and words shared across classes.

Across five languages and six datasets, the method beats soft and automatic verbalizers and approaches fully supervised results with very little data. It also outperforms LLaMA 3.2 (1B) in 0-shot and 4-shot settings. This makes it a cheap, practical option for under-resourced languages.

Key points

  • Data split: 70/30 train/test. From the full dataset, 20% supplies the 4-shot examples and seed words, and 50% is used only to harvest [MASK] predictions. Noisy datasets may use up to five label words per class.
  • Arabic Spam-Ham: XLM-RoBERTa reaches 92.72% accuracy (84.21% macro-F1), while mBERT reaches 90.43% (77.16%).
  • Comparison with other verbalizers: the paper reports gains of 25–30% over Soft and Automatic Verbalizers, although the introduction gives the gain as 10–25%.
  • Other XLM-R accuracies:
    • Urdu Corpus v1: 64.97%, a noisy dataset the authors cite as a limitation
    • iNLTK Urdu News: 89.77%
    • Sindhi: 84.28%
    • Pashto: 39.29%
    • An English dataset (Socher et al., 2013) serves as a left-to-right baseline.
  • LLaMA 3.2 (1B) on Arabic Spam-Ham: it scores 86.08% / 51.41% (accuracy / macro-F1) at 0-shot and 72.83% / 59.40% at 4-shot, but 99.62% / 99.25% with full 70/30 fine-tuning.
  • Code: available at github.com/Furqann-hue/MV.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2ijipj4r77bn

@inproceedings{ullah2026maskedverbalizer,
  title = {{MaskedVerbalizer: Automatic Verbalizer Construction for Few-Shot Text Classification in Low-Resource Right-to-Left Languages}},
  author = {Ullah, Faizad and Sikandar, Furqan and Waqar, Areeba and Ali, Faizan and Ayub, Muhammad Sohaib and Mushtaq, Mubashar and Karim, Asim},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2ijipj4r77bn},
  url = {https://doi.org/10.63317/2ijipj4r77bn}
}

References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meta AI. 2024. LLaMA 3.2: A New Generation of Multilingual Foundation Models. https://ai.meta.com/blog/llama-3- 2- connect -2024-vision-edge-mobile-devices/. Model release; 1B and 3B parameter versions. ↗
  2. Aizaz Ali. 2024. Pashtu Text Sentiment Analysis Dataset. https://doi.org/10.17632/s9k 7dk9sc6.1. Mendeley Data, V1. — [doi:10.17632/s9k](https://doi.org/10.17632/s9k) ↗
  3. Gaurav Arora. 2020. iNLTK: Natural Language Toolkit for Indic Languages. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 66–71, Online. Association for Computational Linguistics. find
  4. Canyu Chen and Kai Shu. 2023. PromptDA: Label-guided Data Augmentation for Prompt-based Few Shot Learners. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 562–574, Dubrovnik, Croatia. Association for Computational Linguistics. find
  5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  6. Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu. 2022. Prototypical Verbalizer for Prompt-based Few-shot Tuning. find
  7. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7014–7024, Dublin, Ireland. Association for Computational Linguistics. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  9. Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. OpenPrompt: An Open-source Framework for Prompt-learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 105–113, Dublin, Irel… find our card
  10. Ali Faheem, Faizad Ullah, Muhammad Hammad, Ahmed Hassan, Muhammad Sohaib Ayub, and Asim Karim. 2026. U-MIRAGE: Benchmarking Chain-of-Thought Reasoning for Urdu Medical QA. In The 2nd Workshop on NLP for Languages Using Arabic Script (AbjadNLP @ EACL 2026). find
  11. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find our card
  12. Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics… find our card
  13. Sanaa Kaddoura and Safaa Henno. 2024. Dataset of Arabic spam and ham tweets. Data in Brief, 52:109904. find
  14. Muhammad Yaseen Khan, Shah Muhammad Emaduddin, and Khurum Nazir Junejo. 2017. find
  15. Harnessing English Sentiment Lexicons for Polarity Detection in Urdu Tweets: A Baseline Approach. In 2017 IEEE 11th International Conference on Semantic Computing (ICSC), pages 242–249. find
  16. Muhammad Yaseen Khan and Muhammad Suffian Nizami. 2020. Urdu Sentiment Corpus (v1.0): Linguistic Exploration and Visualization of Labeled Dataset for Urdu Sentiment Analysis. In 2020 International Conference on Information Science and Communication Technology (ICISCT), pages 1–15. find
  17. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist… find
  18. OpenAI. 2025. GPT-5. https://openai.com/. Large language model. ↗
  19. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5569–5578, Barcelona, Spain (Online). International Committee on C… find
  20. Timo Schick and Hinrich Schütze. 2021. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255– 269, Online. Association for Comp… find our card
  21. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  22. Safdar Soomro, Siti Yuhaniz, Mazhar Dootio, Dr Mujtaba, and Jawaid Siddiqui. 2025. Category-Based Sentiment Analysis of Sindhi News Headlines Using Machine Learning, Deep Learning, and Transformer Models. IEEE Access, PP:1–1. find
  23. Faizad Ullah, Ubaid Azam, Ali Faheem, Faisal Kamiran, and Asim Karim. 2023. Comparing Prompt-Based and Standard Fine-Tuning for Urdu Text Classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6747–6754, Singapore. Association for Computational Linguistics. find
  24. Faizad Ullah, Ali Faheem, Ubaid Azam, Muhammad Sohaib Ayub, Faisal Kamiran, and Asim Karim. 2024. Detecting Cybercrimes in Accordance with Pakistani Law: Dataset and Evaluation Using PLMs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and… find
  25. Faizad Ullah, Safiullah Faizullah, Imdad Ullah Khan, Turki Alghamdi, Toqeer Ali Syed, Ahmad B Alkhodre, Muhammad Sohaib Ayub, and Asim Karim. 2025. Prompt-based fine-tuning with multilingual transformers for language-independent sentiment analysis. Scientific reports, 15(1):20834. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ullah-2026-maskedverbalizer-automatic-verbalizer-construction-few/