Elamur Research⌕
☰
home / papers / 2026 / GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models

GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models

Paper · 2026 · Tianyu Pan and Damon L. Woodard

Summary

Large language models are hard to inspect. Post hoc tools such as saliency maps or sparse autoencoders run into superposition, and hidden states tend to be anisotropic, crowding into a few dominant directions. GeoLAN addresses this during training rather than after it. The authors treat each token's layer-by-layer hidden states as a path, or "tube", through representation space. They borrow the idea of stickiness from the recent proof of the Kakeya conjecture in three dimensions and state two Wolff-style axioms. One limits how many token tubes can pack into a narrow region. The other limits attention heads from collapsing onto the same subspace. A theorem shows that K-sticky models split into nearly orthogonal "grains" with bounded output sensitivity. Two differentiable penalties, KT-CW and KT-Attn, enforce the axioms in practice. On Gemma-3 (1B, 4B, 12B) and Llama-3-8B, GeoLAN frequently keeps task performance intact, yields better geometric measurements, and lessens certain biases. The gains are clearest in mid-sized models. The work points toward interpretability that is built into training rather than reconstructed afterward.

Key points

  • The framework has two regularizers. Katz-Tao Convex Wolff (KT-CW) pushes toward isotropy by using random projections to match projection variance to 1/d. Katz-Tao Attention (KT-Attn) discourages low-rank, collapsed attention routing.
  • The theory is inspired by sticky Kakeya sets (Wang and Zahl, 2025). The Semantic Stickiness Theorem claims a grain decomposition, near-orthogonality bounded by O(K⁻¹), and a Lipschitz robustness bound.
  • The models tested are Gemma-3 1B, 4B and 12B, plus Llama-3-8B. Llama-3-8B and other mid-sized models benefit most. The 1B model can be harmed, and Gemma-3-12B shows an accuracy drop.
  • On CrowS-Pairs, Gemma-3-12B shows a small but statistically significant reduction in stereotype rate, together with a modest decline in clean accuracy.
  • Computing the KT-CW loss slows training throughput by about 15–20%. Code is released on GitHub (GeoLAN_ACL26).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of FloridaGainesville · United States36 works · 364 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Tianyu Pan and Damon L. Woodard. 2026. GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.271

@inproceedings{pan2026geolan,
  title = {{GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models}},
  author = {Tianyu Pan and Damon L. Woodard},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.271},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.271}
}

References

35 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… find
  2. Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations. find
  3. Hila Chefer, Shir Gur, and Lior Wolf. 2021. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 782–791. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  6. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint arXiv:1909.00512. find
  7. Anton Korznikov, Andrey Galichin, Alexey Dontsov, Oleg Y Rogov, Ivan Oseledets, and Elena Tutubalina. 2025. The rogue scalpel: Activation steering compromises llm safety. arXiv preprint arXiv:2509.22067. — [arXiv:2509.22067](https://arxiv.org/abs/2509.22067) find
  8. Bianka Kowalska and Halina Kwaśnicka. 2025. Unboxing the black box: Mechanistic interpretability for algorithmic understanding of neural networks. arXiv preprint arXiv:2511.19265. — [arXiv:2511.19265](https://arxiv.org/abs/2511.19265) find
  9. Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, and Blake A Richards. 2025. Tracing the representation geometry of language models from pretraining to post-training. arXiv preprint arXiv:2509.23024. — [arXiv:2509.23024](https://arxiv.org/abs/2509.23024) find
  10. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252.
  11. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2024. Towards faithful model explanation in nlp: A survey. Computational Linguistics, 50(2):657– 723.
  12. Anemily Machina and Robert Mercer. 2024. Anisotropy is not inherent to transformers. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4892–4907. find
  13. Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc interpretability for neural nlp: A survey. ACM Computing Surveys, 55(8):1–42. find
  14. Ziqing Fan, Siyuan Du, Shengchao Hu, Pingjie Wang, Li Shen, Ya Zhang, Dacheng Tao, and Yanfeng Wang. 2025. Combatting dimensional collapse in llm pre-training data via diversified file selection. arXiv preprint arXiv:2504.20644. — [arXiv:2504.20644](https://arxiv.org/abs/2504.20644) find
  15. Hemant Misra, Shajith Ikbal, Hervé Bourlard, and Hynek Hermansky. 2004. Spectral entropy based feature for robust asr. In 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages I–193. IEEE. find
  16. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1– 42. find
  17. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217.
  18. Larry Guth and Joshua Zahl. 2018. Polynomial wolff axioms and kakeya-type estimates in r 4. Proceedings of the London Mathematical Society, 117(1):192– 220. find
  19. Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. 2024. Llms know more than they show: On the intrinsic representation of llm hallucinations. arXiv preprint arXiv:2410.02707.
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  21. Wenbo Pan, Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. 2025. The hidden dimensions of llm alignment: A multidimensional analysis of orthogonal safety directions. arXiv preprint arXiv:2502.09674. — [arXiv:2502.09674](https://arxiv.org/abs/2502.09674) find
  22. David Klindt, Charles O’Neill, Patrik Reizinger, Harald Maurer, and Nina Miolane. 2025. From superposition to sparse codes: interpretable representations in neural networks. arXiv preprint arXiv:2503.01824. — [arXiv:2503.01824](https://arxiv.org/abs/2503.01824) find
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text 5502 10 transformer. Journal of machine learning research, 21(140):1–67. find
  24. Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646. find
  25. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3505–3506. find
  26. Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. 2024. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 868–874. find
  27. William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff. 2022. Isoscore: Measuring the uniformity of embedding space utilization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3325–3339. find
  28. Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. 2025. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv preprint arXiv:2503.05613. find
  29. Kasia Söderlund. 2025. High-risk ai transparency? on qualified transparency mandates for oversight bodies under the eu ai act. Technology and Regulation, 2025:97–113. find
  30. Kriz Tahimic and Charibeth Cheng. 2025. Mechanistic interpretability of code correctness in llms via sparse autoencoders. arXiv preprint arXiv:2510.02917. — [arXiv:2510.02917](https://arxiv.org/abs/2510.02917) find
  31. Yuanbo Tang, Zhiyuan Peng, and Yang Li. 2023. Explainable trajectory representation through dictionary learning. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, pages 1–4. find
  32. Hong Wang and Joshua Zahl. 2025. Volume estimates for unions of convex sets, and the kakeya set conjecture in three dimensions. arXiv preprint arXiv:2502.17655. — [arXiv:2502.17655](https://arxiv.org/abs/2502.17655) find
  33. Thomas Wolff. 1995. An improved bound for kakeya type maximal functions. Revista Matemática Iberoamericana, 11(3):651–674. find
  34. Joshua Zahl. 2025. A survey of the kakeya conjecture, 2000-2025. arXiv preprint arXiv:2512.09397. find
  35. Jiahuan Zhang, Tianheng Wang, Ziyi Huang, Yulong Wu, Hanqing Wu, DongbaiChen DongbaiChen, Linfeng Song, Yue Zhang, Guozheng Rao, and Kaicheng Yu. 2025. Sr-llm: Rethinking the structured representation in large language model. In Proceedings 5503 11 of the 63rd Annual Meeting of the Association for C…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-geolan-geometric-learning-latent-explanatory/