Elamur Research⌕
☰
home / papers / 2026 / Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization

Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization

Paper · 2026 · Yurui Pan, Ke Xu and Bo Peng

Summary

Common LLM alignment objectives (SFT, RLHF, DPO) optimise token likelihoods or scalar preference scores. They place no constraint on how hidden representations move from a prompt to a response. The authors treat generation as a path through hidden space and regularise that path with 0-dimensional persistent homology.

For SFT, Trajectory Topology Loss (TTL) puts a batch's mean-pooled prompt and gold-answer embeddings into one point cloud. A Union-Find procedure then finds the merge edges that first connect prompt and answer components. A cosine term pushes the model's prompt-to-answer shift toward these "bridges".

For DPO, Topological Preference Optimization (TPO) pulls the rejected-to-chosen hidden-state difference at an intermediate layer toward topic-specific preference vectors. These vectors are built offline through clustering and LLM labelling, and an EMA scheme balances the DPO and TPO terms.

On Qwen2.5-7B-Instruct with UltraChat and Anthropic HH-RLHF, both methods beat plain baselines and non-topological regularisers without worsening toxicity. This suggests that inexpensive topological signals can make alignment more controllable. However, only one model and English data were tested.

Key points

  • Base model is Qwen2.5-7B-Instruct. UltraChat is used for SFT experiments, and Anthropic HH-RLHF is used for DPO experiments.
  • On HH-RLHF, TPO improves on plain DPO: RewardBench rises from 84.5 to 87.2, AlpacaEval win rate from 52.1% to 55.4%, and harmlessness from 90.2% to 93.5%.
  • Topic-aware preference vectors outperform a single global vector (RM 87.2 vs 85.1; helpfulness 8.81 vs 8.68). Removing EMA dynamic weighting lowers results.
  • The full pipeline (SFT+TTL, then DPO+TPO) reaches RewardBench 88.1, AlpacaEval 56.5%, MT-Bench 8.88 and harmlessness 94.5%. The SFT+DPO baseline scores 84.5, 52.1%, 8.65 and 90.2% on the same metrics.
  • Ablations show that persistent-homology bridges beat random, per-example and nearest-neighbour pairings. Limitations include English-only evaluation, LLM-based topic labelling, and pairwise-distance cost that grows quadratically with batch size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓Tongji UniversityShanghai · China19 works · 2,162 citations

Also on the byline: Shanghai Ocean University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yurui Pan, Ke Xu and Bo Peng. 2026. Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1242

@inproceedings{pan2026topology,
  title = {{Topology-Enhanced Alignment for Large Language Models: Trajectory Topology Loss and Topological Preference Optimization}},
  author = {Yurui Pan, Ke Xu and Bo Peng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1242},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1242}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alessandro Achille and Stefano Soatto. 2018. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50):1–54. find
  2. Henry Adams, Trey Emerson, Martin Kirby, Christopher J. Neville, Chris Peterson, Patrick Shipman, Svetlana Chepushtanova, Mariah Hanson, Fabio Motta, and Leonard Ziegelmeier. 2015. Persistence images: A stable vector representation of persistent homology. Journal of Machine Learning Research, 18(8):… find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  4. Rubén Ballester, Carles Casacuberta, and Sergio Escalera. 2024. Topological data analysis for neural network analysis: A comprehensive survey. arXiv preprint arXiv:2312.05840. — [arXiv:2312.05840](https://arxiv.org/abs/2312.05840) find
  5. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3319–3327. find
  6. Ralph Allan Bradley and Milton E. Terry. 1952. The rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39:324– 345. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  8. Peter Bubenik. 2015. Statistical topological data analysis using persistence landscapes. Journal of Machine Learning Research, 16(1):77–102. find
  9. Gunnar Carlsson. 2009. Topology and data. Bulletin of the American Mathematical Society, 46(2):255–308. find
  10. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706.03741. find
  11. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051.
  12. Herbert Edelsbrunner and John Harer. 2010. Computational topology: An introduction. American Mathematical Society. find
  13. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of EMNLP-IJCNLP, pages 55–65. find
  14. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3356–3369.
  15. Robert Ghrist. 2008. Barcodes: the persistent topology of data. Bulletin of the American Mathematical Society, 45(1):61–75. find
  16. Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations. find
  17. Christoph Hofer and 1 others. 2019. Deep learning with topological signatures. In NeurIPS. find
  18. Joseph Kruskal. 1956. On the shortest spanning subtree of a graph. Proc. AMS. find
  19. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, volume 31, pages 7167–7177. find
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the ICLR 2013 Workshop on Representation Learning for NLP.
  21. Guillermo Ortiz-Jiménez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2020. Neural anisotropy directions. arXiv preprint arXiv:2006.09717. — [arXiv:2006.09717](https://arxiv.org/abs/2006.09717) find
  22. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  23. Robin L. Plackett. 1975. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193–202. find
  24. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  25. Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. 2017. On the expressive power of neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 2847–2854. find
  26. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of EMNLP-IJCNLP, pages 3982– 3992.
  27. K. Rieck, H. Leitte, C. Höfer, and H. Wagner. 2019. Neural persistence: A complexity measure for deep neural networks using algebraic topology. In International Conference on Learning Representations. find
  28. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. find
  29. Robert Endre Tarjan. 1975. Efficiency of a good but not linear set union algorithm. Journal of the ACM, 22(2):215–225. find
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-topology-enhanced-alignment-large-language/