Elamur Research⌕
☰
home / papers / 2026 / The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models

The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models

Paper · 2026 · Reem I. Masoud, Chen Feng, Shunta Asano et al.

Summary

The paper asks whether the linguistic makeup of fine-tuning data can tell us, before any training, how well a model will handle culture-specific tasks. The authors take a dataset-centric approach. They describe Arabic, Chinese and Japanese post-training datasets with cheap metrics covering lexical diversity, lexical richness, semantic similarity and cluster cohesion. They then run PCA separately for each language. The first three components are fairly interpretable: semantic structure, then diversity and lexical variation, then a more language-specific axis. Next, LLaMA, Mistral and DeepSeek models are fine-tuned with QLoRA on each full dataset and on controlled subsets, then evaluated on benchmarks that measure what models know about cultures as well as their grasp of culturally grounded values and social norms. The component scores do relate to downstream results, but no single component predicts performance everywhere. Size-matched subset selection guided by the components can beat random sampling, but only once the right component and direction have been checked for a given language and model. PC3 works best for Arabic and High-PC1 for Japanese, especially with LLaMA. Chinese results remain tentative. The work matters because it supports cheap data diagnostics before training, while warning against one universal quality criterion.

Key points

  • Feature set: Distinct-1, Distinct-2, Self-BLEU, TTR, MATTR, MTLD, HDD, cosine similarity over TF–IDF and sentence-transformer embeddings, and the silhouette score of K-means clusters. All are computed on 1,000 randomly sampled examples per dataset.
  • Features are standardized within each language, and one PCA per language keeps three components. Pearson correlations link dataset scores to benchmark results and are treated as descriptive, not causal.
  • Subset intervention: for each language and component, the five top-scoring datasets are split into HIGH-PC, LOW-PC and size-matched RANDOM subsets. Every run starts fresh from the same checkpoint with fixed preprocessing and QLoRA settings.
  • Margins relative to random vary widely, which shows strong model dependence:
    • Databricks-Dolly-JA (PC2-L) gives +0.298 for DeepSeek.
    • COIG-PC (PC1-L) gives +0.244 for LLaMA.
    • Open-ArabicaQA (PC1-H) gives −0.302 for DeepSeek.
  • In Arabic, PC1 brings big gains for LLaMA and Mistral but large losses for DeepSeek. PC2 is mostly weak or harmful.

Interesting ideas

  • I1. Normalize metrics inside each language before reducing dimensions, so differences in script, morphology and tokenization don't swamp the comparison.
  • I2. Since PCA works on whole datasets, the authors build a per-example ranking score by weighting a few per-example features with the component loadings. This turns a corpus-level signal into a filter for individual items.
  • I3. Every HIGH and LOW selection is compared against a size-matched RANDOM subset drawn from unused examples with a fixed seed. Without this, a filter could look useful merely because it changes data volume.
  • I4. Results are reported both as win/tie/loss counts across benchmarks and as a normalized average margin. This separates small effects that recur often from rare large swings.
  • I5. The same data-selection direction can help one architecture and hurt another, so "high" and "low" ends are best seen as poles of an axis, not as good and bad data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when comparing silver corpora or treebanks from different sources or genres, standardize features within each group before ranking them, so source quirks don't look like quality.
  • I2 (learn): weight cheap per-sentence features (parser agreement, length, lexical diversity) by corpus-level loadings to rank silver sentences for training.
  • I3 (learn): treat a size-matched random sample as the required negative control whenever a new silver-data filter or repair-rule set is evaluated.
  • I4 (answer): score reader changes on school tests and fable QA with both views, so one big jump on a single test set isn't mistaken for general progress.
  • I5 (tree): pick training treebanks and subsets separately for the perceptron parser and gparse, rather than assuming data that helps one will help the other.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksSemitic languages · 1,484 works

Cite

Cite as Reem I. Masoud, Chen Feng, Shunta Asano, Saied Alshahrani, Philip Colin Treleaven and Miguel R. D. Rodrigues. 2026. The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models. arXiv:2602.01161.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2602.01161 · arXiv 2602.01161

@misc{masoud2026role,
  title = {{The Role of Dataset Linguistic Structure in the Cultural Awareness of Large Language Models}},
  author = {Reem I. Masoud, Chen Feng, Shunta Asano, Saied Alshahrani, Philip Colin Treleaven and Miguel R. D. Rodrigues},
  year = {2026},
  doi = {10.48550/arxiv.2602.01161},
  eprint = {2602.01161},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2602.01161}
}

References

57 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. llm-book/aio: Ai king competition qa dataset. HuggingFace Dataset. find
  2. Abdelrahman Abdallah, Mahmoud Kasem, Mahmoud Abdalla, Mohamed Mahmoud, Mohamed Elkasaby, Yasser Elbendary, and Adam Jatowt. 2024. Arabicaqa: A comprehensive dataset for arabic question answering. Preprint, arXiv:2403.17848. ↗
  3. Ahmed Hashim Al-Dulaimi. 2022. Ultimate Arabic News Dataset. ↗
  4. Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. arXiv preprint arXiv:2402.13231. ↗
  5. Saied Alshahrani, Norah Alshahrani, Soumyabrata Dey, and Jeanna Matthews. 2023. Performance implications of using unrepresentative corpora in arabic natural language processing. In Proceedings of Arabic-NLP 2023, pages 218–231. find
  6. Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran AQ Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, et al. 2024. Cidar: Culturally relevant instruction dataset for arabic. arXiv preprint arXiv:2402.03177. ↗
  7. ArbML. 2023a. Al jazeera news articles. https://huggingface.co/datasets/arbml/news_articles_aljazeera. HuggingFace Dataset. ↗
  8. ArbML. 2023b. Arsen-20. https://huggingface.co/datasets/arbml/Arsen_20. HuggingFace Dataset. ↗
  9. Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Ziqiang Liu, Junting Zhou, Tianyu Zheng, Xincheng Zhang, Nuo Ma, Zekun Wang, et al. 2024. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. Preprint, arXiv:2403.18058. ↗
  10. Francis Bond and Takayuki Kuribayashi. 2023. The Japanese Wordnet 2.0. In Proceedings of the 12th Global Wordnet Conference, pages 179–186, University of the Basque Country, Donostia - San Sebastian, Basque Country. Global Wordnet Association. find
  11. Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. 2024. On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410.15226. ↗
  12. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. 2024. Culturalbench: A robust, diverse, and challenging cultural benchmark by human-ai culturalteaming. arXiv preprint arXiv:2410.02677. ↗
  13. Michael A Covington and Joe D McFall. 2010. Cutting the gordian knot: The moving-average type–token ratio (mattr). Journal of quantitative linguistics, 17(2):94–100. ↗
  14. Iris Dominguez-Catena, Daniel Paternain, and Mikel Galar. 2025. Dsap: Analyzing bias through demographic comparison of datasets. Information Fusion, 115:102760. ↗
  15. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369. ↗
  16. Md Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2024. Nativqa: Multilingual culturally-aligned natural query for llms. arXiv preprint arXiv:2407.09823. ↗
  17. Masanori HIRANO, Masahiro SUZUKI, and Hiroki SAKAJI. 2023. llm-japanese-dataset v0: Construction of Japanese Chat Dataset for Large Language Models and its Methodology. ↗
  18. Geert Hofstede, Gert Jan Hofstede, and Michael Minkov. 2013. Vsm 2013 — values survey module 2013. https://geerthofstede.com/research-and-vsm/vsm-2013/. Accessed: 2025-01-04. ↗
  19. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, et al. 2023a. Acegpt, localizing large language models in arabic. arXiv preprint arXiv:2309.12053. ↗
  20. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023b. Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322. ↗
  21. Walaa Saber Ismail and Masun Nabhan Homsi. 2018. Dawqas: A dataset for arabic why question answering system. Procedia computer science, 142:123– 131. find
  22. Izumi Lab. 2023. wikipedia-ja-20230720. find
  23. Webdell Johnson. 1944. Studies in language behavior: A program of research. Psychological Monographs, 56(2):1–15. find
  24. Elise Karinshak, Amanda Hu, Kewen Kong, Vish-wanatha Rao, Jingren Wang, Jindong Wang, and Yi Zeng. 2024. Llm-globe: A benchmark evaluating the cultural values embedded in llm output. arXiv preprint arXiv:2411.06032. ↗
  25. Akihiro Katsuta and Kazuhide Yamamoto. 2018. Crowdsourced corpus of sentence simplification with core vocabulary. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC), pages 461–466. find
  26. "Fajri Koto, Haonan Li, Sara Shatanawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin". 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. In Findings of the Association for Computational Linguistics: ACL 2024.
  27. Kunishou. 2023. Databricks dolly 15k (japanese). https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja. HuggingFace Dataset. levelevel. 2023. Aozoratxt: Aozora bunko text corpus. https://github.com/levelevel/AozoraTxt. GitHub Repository. ↗
  28. Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024. Culturepark: Boosting cross-cultural understanding in large language models. Advances in Neural Information Processing Systems, 37:65183–65216. find
  29. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 110–119. ↗
  30. Reem I Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2023. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. arXiv preprint arXiv:2309.12342. ↗
  31. Philip M McCarthy and Scott Jarvis. 2010. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392. ↗
  32. James B McQueen. 1967. Some methods of classification and analysis of multivariate observations. In Proc. of 5th Berkeley Symposium on Math. Stat. and Prob., pages 281–297. find
  33. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2024. Aradice: Benchmarks for dialectal and cultural capabilities in llms. arXiv preprint arXiv:2409.11404. ↗
  34. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2023. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456. ↗
  35. Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. 2024. Improving data efficiency via curating llm-driven rating systems. arXiv preprint arXiv:2410.10877. ↗
  36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  37. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pages 1–96. ↗
  38. Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. arXiv preprint arXiv:2211.13069. ↗
  39. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992. ↗
  40. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53–65. ↗
  41. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Cahtrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in arab culture. Preprint, arXiv:2502.12788. ↗
  42. Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513– 523. ↗
  43. Gerard Salton, Anita Wong, and Chung-Shu Yang. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11):613–620. ↗
  44. Satoshi Sekine, Maya Ando, Michiko Goto, Kumi Suzuki, Daisuke Kawahara, Naoya Inoue, and Kentaro Inui. 2024. Creation of japanese instruction data for llms: ichikara-instruction. Proceedings of the 30th Annual Meeting of the Association for Natural Language Processing, pages 1508–1513. (in Japanese). find
  45. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, et al. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  46. Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. Preprint, arXiv:2402.06619. ↗
  47. ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022. Jaquad: Japanese question answering dataset for machine reading comprehension. arXiv preprint arXiv:2202.01764. ↗
  48. Masashi Takeshita and Kenji Araki. 2023. Jcommonsensemorality. In Proceedings of the 29th Annual Meeting of the Association for Natural Language Processing (in Japanese), pages 357–362. find
  49. Masashi Takeshita and Rafal Rzepka. 2025. Jethics: Japanese ethics understanding evaluation dataset. Preprint, arXiv:2506.16187. ↗
  50. Wikimedia Foundation. 2024. Wikimedia project dumps. https://dumps.wikimedia.org/. Accessed 2024. ↗
  51. Wikinews. 2024. Japanese wikinews. https://ja.wikinews.org/wiki/. Accessed 2024. ↗
  52. Trong Wu. 1993. An accurate computation of the hypergeometric distribution function. ACM Transactions on Mathematical Software (TOMS), 19(1):33–43. ↗
  53. Gregory Yauney, Emily Reif, and David Mimno. 2023. Data similarity is not enough to explain language model performance. arXiv preprint arXiv:2311.09006. ↗
  54. Linhao Yu, Yongqi Leng, Yufei Huang, Shang Wu, Haixin Liu, Xinmeng Ji, Jiahui Zhao, Jinwang Song, Tingting Cui, Xiaoqing Cheng, Tao Liu, and Deyi Xiong. 2024. Cmoraleval: A moral evaluation benchmark for chinese large language models. Preprint, arXiv:2408.09819. ↗
  55. Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. 2025. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25949– 25957. find
  56. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. arXiv preprint arXiv:2404.16308. ↗
  57. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/masoud-2026-role-dataset-linguistic-structure-cultural/