Elamur Research⌕
☰
home / papers / 2026 / Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation

Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation

Paper · 2026 · Haziq Khalid, Salsabeel Shapsough and Imran Zualkernan

Summary

The paper tackles a tension in automatic story writing for Arabic Early Grade Reading Assessment (EGRA). Passages must respect strict limits on length, vocabulary, reading level, narrative structure and cultural neutrality, yet repetitive plots weaken their value as assessment items. The authors focus on small 7–9B models, which suit low-resource deployment but become unstable when pushed toward variety. They propose noise steering, which adds calibrated Gaussian perturbations to internal transformer states during inference without any fine-tuning. Four injection sites are compared: token embeddings, the attention output branch, the residual stream after a full block, and an entropy-gated attention variant (AENI). These are tested against high-temperature sampling baselines on five Arabic-centric and multilingual models. Residual stream noise (L-RES) and AENI raise narrative diversity while keeping quality, constraint adherence and early-grade readability statistically close to the baseline. High-temperature sampling, by contrast, pushes vocabulary toward older grade levels and makes some models collapse. The main takeaway is that perturbing internal representations is a safer route to variety than adding randomness at the output when content is tightly constrained.

Key points

  • Four noise strategies: embedding noise, attention-logit noise (added after W_O, before the MLP, KV cache untouched), residual stream noise (L-RES), and AENI. AENI scales noise by how peaked attention is, in the spirit of EDT.
  • Noise is applied only during decoding, never to the prompt prefill. Its magnitude follows a cosine decay over generation, with a single calibration value α = 0.175 used for all methods.
  • Models include Jais, ALLaM, Fanar, AceGPT and Phi-4. Diversity is measured with Self-BLEU, semantic dispersion and Vendi Score; quality and part of constraint adherence are judged by GPT, alongside programmatic checks for word count and tense.
  • High-temperature baselines (T = 1.8, k = 40, p = 0.9) helped only Jais, raising its Vendi Score from 4.71 to 6.37 or 5.87. Elsewhere, ALLaM quality fell to about 5 and Fanar to about 4, AceGPT and Phi-4 collapsed, and reading level drifted from Grade 3 to Grades 5–6.
  • AENI stabilizes raw attention noise at a small cost in diversity. On AceGPT the two give nearly the same Vendi Score (10.05 vs 10.21).
  • Models with stronger baseline quality absorb noise more gracefully. The authors therefore frame noise steering as a way to unlock latent diversity, not to make up for weak language ability.

Interesting ideas

  • I1. Strong early perturbation that fades over the course of generation lets the opening of a story explore while the ending returns to following the constraints, a useful schedule wherever creativity must coexist with rules.
  • I2. The instruction and prompt stay clean and only the newly generated part is perturbed, which keeps the model's reading of the constraints intact.
  • I3. The paper separates useful variation (a different plot) from harmful variation (harder vocabulary) and measures reading grade level as a separate guardrail. A diversity method is accepted only if the readability metric does not move.
  • I4. Perturbation is triggered by the system's own overconfidence, here peaked attention entropy, so exploration happens only where the system is stuck in a rut.
  • I5. To reduce relative-scoring bias, outputs were judged in fixed-size batches with all experimental conditions shuffled together, so the judge could not compare runs side by side.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When generating text from a meaning graph, Mova could vary choices (event order, referring expressions) strongly at the start of a text and narrow them toward the end so closing sentences stay faithful to the graph.
  • I2 (back): Keep the source world and its frames fixed and deterministic, and apply randomized variation only within the surface realization stage so diversity never corrupts the meaning being expressed.
  • I3 (back): Treat vocabulary level as an explicit constraint on generation and paraphrase output, checking grade level against a negative control so that varied wording does not quietly drift toward adult vocabulary.
  • I4 (learn): In self-play or silver-data generation, Mova could add variation only on cases where all witness parsers or rules agree with very low uncertainty, broadening coverage without destabilizing already varied cases.
  • I5 (learn): When an LLM teacher grades Mova's outputs or competing analyses, mixing conditions into shuffled fixed batches would make the teacher's scores more comparable across experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University of SharjahSharjah · United Arab Emirates4 works · 81 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksText generation · 202 works

Cite

Cite as Haziq Khalid, Salsabeel Shapsough and Imran Zualkernan. 2026. Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.41

@inproceedings{khalid2026noise,
  title = {{Noise Steering for Controlled Text Generation: Improving Diversity and Reading-Level Fidelity in Arabic Educational Story Generation}},
  author = {Haziq Khalid, Salsabeel Shapsough and Imran Zualkernan},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.41},
  url = {https://doi.org/10.18653/v1/2026.bea-1.41}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Abdelghafur, Y. Kaddoura, S. Shapsough, I. Zualkernan, and E. Kochmar. 2025. Tales from the algorithm: Enhancing reading comprehension assessments with ai-generated arabic stories. In Proceedings of the 10th International Congress on Information and Communication Technology (ICICT 2025), volume 1443 of Lecture Notes in Networks and Systems, pages 277–288. find
  2. David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science, 9(1):147–169. ↗
  3. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Majed Alrubaian, Ali Alammari, Zaki Alawami, and Abdulmohsen Al-Thubaity. 2024. ALLAM: Large language models for Arabic and English. arXiv preprint arXiv:2407.15390. ↗
  4. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, Miami, Florida, USA. Association for Computational Linguistics.
  5. Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, and Masoomali Fatehkia. 2025. Fanar: An Arabic-centric multimodal generative AI platform. arXiv preprint arXiv:2501.13944. ↗
  6. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, volume 35. find
  7. Jacob Dineen, Aswin RRV, Zhikun Xu, and Ben Zhou. 2026. Vocabulary dropout for curriculum diversity in llm co-evolution. Preprint, arXiv:2604.03472. ↗
  8. Margaret M. Dubeck and Amber Gove. 2015. The early grade reading assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development, 40:315–322. ↗
  9. Mahmoud El-Haj and Paul Rayson. 2016. OS-MAN — a novel Arabic readability metric. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 250–255, Portorož, Slovenia. European Language Resources Association (ELRA). find
  10. Ahmed O. El-Shangiti and 1 others. 2024. Arabic automatic story generation with large language models. In Proceedings of the First Workshop on Arabic Natural Language Processing.
  11. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. find
  12. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. ↗
  13. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Dan Friedman and Adji Bousso Dieng. 2023. The vendi score: A diversity evaluation metric for machine learning. Preprint, arXiv:2210.02410. ↗
  14. Nizar Y. Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. ↗
  16. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics. ↗
  17. Minki Kang, Sung Ju Hwang, Gibbeum Lee, and Jaewoong Cho. 2024. Latent paraphrasing: Perturbation on layers improves knowledge injection in language models. In Advances in Neural Information Processing Systems, volume 37. find
  18. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems. find
  19. Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. Locally typical sampling. Transactions of the Association for Computational Linguistics, 11:102–121. ↗
  20. Minh Nhat Nguyen, Andrew Baker, Clement Neo, Allen Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. 2025. Turning up the heat: Min-p sampling for creative and coherent llm outputs. Preprint, arXiv:2407.01082. ↗
  21. OpenAI. 2026. GPT-5.3 Instant ChatGPT model. Accessed: 2026-03-06. find
  22. Samarth Rai, Salsabeel Shapsough, and Imran Zualkernan. 2024. Measuring fluency, coherency and logicality of GPT-4 generated EGRA comprehension stories. In Proceedings of the 2024 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 201–203. find
  23. RTI International. 2016. Early grade reading assessment toolkit. Technical report, RTI International / USAID. find
  24. Shimao Zhang, Yu Bao, and Shujian Huang. 2024. EDT: Improving large language models’ generation by entropy-based dynamic temperature sampling. arXiv preprint arXiv:2403.14541. ↗
  25. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1097– 1100. ↗
  26. Imran Zualkernan and Salsabeel Shapsough. 2024. Towards using large language models to automatically generate reading comprehension assessments for early grade reading assessment. pages 3772–3782. find
  27. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, and 3 others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149. ↗
  28. Prithviraj Singh Shahani and 1 others. 2025. Noise injection systemically degrades large language model safety guardrails. arXiv preprint arXiv:2505.13500. ↗
  29. Aadhith Shankarnarayanan, Taufiq Syed, Salsabeel Y. Shapsough, and Imran A. Zualkernan. 2024. Once upon a GPT-4: Enhancing diversity in automated reading comprehension story generation with classic tales. In Proceedings of the 2024 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 196–200. find
  30. Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762. ↗
  31. Brandon T. Willard and Rémi Louf. 2023. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalid-2026-noise-steering-controlled-text-generation/