Elamur Research⌕
☰
home / papers / 2026 / Far Out: Evaluating Language Models on Slang in Australian and Indian English

Far Out: Evaluating Language Models on Slang in Australian and Indian English

Paper · 2026 · Deniz Kaya Dilsiz, Dipankar Srirag and Aditya Joshi

Summary

The paper asks whether current language models understand regional slang beyond Standard American English. It compares Australian English (en-AU), mostly a first language, with Indian English (en-IN), often an additional language. The authors took slang candidates from Wiktionary, pulled definitions and example sentences from Urban Dictionary, and had a native-speaker expert per variety filter them. The result is WEB, with 377 usage examples. They then prompted Gemini Pro 2.5 to write varied scenarios for each phrase, producing GEN, with 1,492 examples. The slang phrase is masked, and seven encoder and decoder models must either produce it or pick it from candidates. Open-ended prediction almost always fails, while multiple-choice selection works far better. This suggests models can recognise slang more easily than they can produce it. en-IN items are handled better than en-AU items. A study of errors shows models often grasp the meaning but swap in literal or generic wording. The work offers a reusable benchmark and shows that variety-specific vocabulary remains a weak spot even for English.

Key points

  • Data filtering: Source collection gave 940 en-IN and 2,540 en-AU candidates. After expert filtering, 64 en-IN and 313 en-AU phrases remained in WEB. GEN adds four Gemini-written scenarios per phrase, giving 248 en-IN and 1,244 en-AU examples.
  • Three tasks:
    • open target word prediction (TWP);
    • a guided variant (TWP∗);
    • four-way target word selection (TWS), with three random distractors from the same variety.
  • Models: BERT-Base, RoBERTa-Large, XLM-RoBERTa-Large, Granite-4.0-1B, Llama-3.2-3B-Instruct, Olmo-2-7B-Instruct and Qwen3-4b-Instruct, ranging from 110M to 8B parameters. All were run 8-bit quantized on a laptop.
  • Main results:
    • Average accuracy rises from 0.03 on TWP to 0.49 on TWS.
    • Open prediction never exceeds 0.13 average accuracy, while selection reaches up to 0.8.
    • On TWS, en-IN outperforms en-AU (0.54 vs 0.44).
    • WEB yields slightly higher similarity scores than GEN, by 0.03 on TWP and 0.05 on TWP∗.
  • Dataset diagnostics:
    • GPT-2 perplexity is far higher on WEB (701.5 en-AU, 406.5 en-IN) than on GEN (about 47–49).
    • GEN scenarios are diverse, with mean MiniLM cosine similarity of about 0.38 and low ROUGE overlap.
    • The Sentence-BERT similarity metric agrees with Granite-embedding-125m-english at a Pearson correlation of 0.77.

Interesting ideas

  • I1. Testing the same item both as free generation and as a choice among options separates knowing a word from producing it, exposing a gap that a single format would hide.
  • I2. Before trusting LLM-written examples, the authors checked how lexically and semantically different the scenarios for each item were, so the benchmark is not just repeating itself.
  • I3. The soft similarity metric was validated by recomputing it with a second, independent embedding model and correlating the two.
  • I4. The error taxonomy sorts mistakes into five types (rendering the slang literally, replacing it with a vague general term, shifting away from the intended meaning, misreading the surrounding context, and outright failure), which separates "meaning understood, wrong form" from real misunderstanding.
  • I5. Comparing perplexity of authentic web examples with synthetic ones shows how noisy and unfluent real informal text is compared with generated text.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test readers could be scored in both open and multiple-choice form, to tell missing knowledge apart from failure to retrieve the right surface form.
  • I2 (learn): Sentences from LLM teacher tiers could pass a pairwise ROUGE plus embedding-similarity filter, keeping near-duplicate silver data out of training.
  • I3 (learn): Any embedding-based score in Mova's evaluation could be paired with a second scorer as a sanity check, much like a witness-agreement test.
  • I4 (world): Compiled knowledge could map variety-tagged slang such as "prepone" or "far out" to ontology concepts, so the world model keeps the meaning while knowing the regional form.
  • I5 (tree): Perplexity could flag noisy, colloquial input that is likely to break the parser, routing it to dialect-aware conversion or extra caution.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓UNSW SydneySydney · Australia36 works · 1,509 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 works

Cite

Cite as Deniz Kaya Dilsiz, Dipankar Srirag and Aditya Joshi. 2026. Far Out: Evaluating Language Models on Slang in Australian and Indian English. Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.vardial-1.2

@inproceedings{dilsiz2026far,
  title = {{Far Out: Evaluating Language Models on Slang in Australian and Indian English}},
  author = {Deniz Kaya Dilsiz, Dipankar Srirag and Aditya Joshi},
  booktitle = {Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects},
  year = {2026},
  doi = {10.18653/v1/2026.vardial-1.2},
  url = {https://doi.org/10.18653/v1/2026.vardial-1.2}
}

References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Parul Awasthy, Aashka Trivedi, Yulong Li, Mihaela Bornea, David Cox, Abraham Daniels, Martin Franz, Gabe Goodhart, Bhavani Iyer, Vishwajeet Kumar, Luis Lastras, Scott McCarley, Rudra Murthy, Vignesh P, Sara Rosenthal, Salim Roukos, Jaydeep Sen, Sukriti Sharma, Avirup Sil, and 3 others. 2025. Granite embedding models. Preprint, arXiv:2502.20204. ↗
  2. Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2025. Natural language processing for dialects of a language: A survey. ACM Comput. Surv., 57(6). ↗
  3. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3416 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261. ↗
  4. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575–3585, Online. Association for Computational Linguistics.
  5. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. ↗
  8. Nicholas Deas, Jessica Grieser, Shana Kleiner, Desmond Patton, Elsbeth Turcan, and Kathleen McKeown. 2023. Evaluation of African American language bias in natural language generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6805– 6824, Singapore. Association for Computational Linguistics. find
  9. Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, and Xueqi Cheng. 2024. SLANG: New concept comprehension of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12558–12575, Miami, Florida, USA. Association for Computational Linguistics. find
  10. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 24 others. 2025. 2 olmo 2 furious. Preprint, arXiv:2501.00656. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  12. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  13. Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or Abhinav Sukumar Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2025. NormAd: A framework for measuring the cultural adaptability of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2373–2403, Albuquerque, New Mexico. Association for Computational Linguistics.
  14. Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dilsiz-2026-far-out-evaluating-language-models/