Elamur Research⌕
☰
home / papers / 2026 / Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.

Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.

Summary

The authors ask whether Transformers can serve corpus linguistics as an exploratory, descriptive instrument rather than only as predictors, and whether such models capture Saussure's paradigmatic relations (substitutable words held in memory) and syntagmatic relations (word order and co-presence in the discourse chain). They train a deliberately small, interpretable GPT-style decoder on a corpus of French political speeches for next-word text generation. They then extract two kinds of features. A global method aggregates attention scores over the whole training corpus into a vocabulary-by-vocabulary matrix, treats its rows as word embeddings, and visualises them with PCA. A local method applies gradient-based saliency to individual generated predictions to find which prompt words drove each output. Both are checked against classical corpus-statistics baselines. According to the authors, the two methods reveal both paradigmatic and syntagmatic phenomena, and the statistical comparisons empirically corroborate these findings. This makes Transformers a credible heuristic tool for discourse analysis, especially for politically oriented corpora.

Key points

  • The model adapts Apoorv Nandan's (2020) Keras "miniature GPT", originally demonstrated on IMDB review generation. It is a single Transformer decoder block with multiple attention heads, a causal mask, positional embeddings, and word-level tokenization manually tailored to the French corpus.
  • The context window is fixed at 20 words, matching the corpus's average sentence length. Attention is summed across heads, giving a 20 × 20 matrix per sequence.
  • For the global method, attention directionality is neutralised by averaging a_mn and a_nm, following Saponati et al. (2025). Scores are then accumulated over all training sequences into a triangular co-occurrence matrix A (m × n, the vocabulary size).
  • The global method draws on attention visualisation (BertViz, Vig 2019), while the local method takes its cue from saliency techniques developed for images, as in the work of Zhou et al. (2016) and Simonyan et al. (2014).
  • The work is positioned as corpus-driven research (Tognini-Bonelli, 2002), aimed at interpretability rather than predictive accuracy. The authors validate the extracted patterns against standard statistics from corpus linguistics (Oakes, 1998).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3tgpydvmvpeu

@inproceedings{vanni2026explore,
  title = {{Explore Political Discourse with Transformers. Emergent Paradigmatic and Syntagmatic Representations.}},
  author = {Vanni, Laurent and Mayaffre, Damon},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3tgpydvmvpeu},
  url = {https://doi.org/10.63317/3tgpydvmvpeu}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Abbas. 2019. Super-hyperbolic man: Hyperbole as an ideological discourse strategy in trump’s speeches. International Journal for the Semiotics of Law, 32:1–18. find
  2. Gavin Abercrombie and Riza Batista-Navarro. 2020. ParlVote: A corpus for sentiment analysis of political debates. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5073–5078, Marseille, France. European Language Resources Association. find
  3. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R.A. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science 41, pages 391–407. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. J. R. Firth. 1957. A synopsis of linguistic theory 1930-55. 1952-59:1–32. find
  6. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2024. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1), pages 16410–16430, Bangkok, Thailand. Associa… find
  7. Zellig Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
  8. Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019. Towards understanding neural machine translation with word importance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  9. Pere-Lluís Huguet Cabot, Verna Dankers, David Abadi, Agneta Fischer, and Ekaterina Shutova. 2020. The Pragmatics behind Politics: Modelling Metaphor, Framing and Emotion in Political Discourse. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4479– 4488, Online. Associ… find our card
  10. Sarthak Jain and Byron C. Wallace. 2019. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543–3556, Minneapolis, Minnesota. Associa… find our card
  11. Charlott Jakob, Pia Wenzel, Salar Mohtaj, and Vera Schmitt. 2024. Augmented political leaning detection: Leveraging parliamentary speeches for classifying news articles. In Proceedings of the 4th Workshop on Computational Linguistics for the Political and Social Sciences: Long and short papers, page… find
  12. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. find our card
  13. Gabriella Lapesa, Stefan Evert, and Sabine Schulte im Walde. 2014. Contrasting syntagmatic and paradigmatic relations: Insights from distributional semantic models. In Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014), pages 160– 170, Dublin, Ireland. Assoc… find
  14. Zachary C. Lipton. 2018. The mythos of model interpretability. Commun. ACM, 61(10):36–43. find
  15. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Neural Information Processing Systems. find our card
  16. Tshilidzi Marwala. 2023. Natural Language Processing in Politics, pages 99–115. Springer Nature Singapore, Singapore. find
  17. Damon Mayaffre and Céline Poudat. 2013. Quantitative approaches to political discourse: corpus linguistics and text statistics. In Kjersti Flottum, editor, Speaking of Europe. Approaches to complexity in European political discourse, pages 65–83. John Benjamins. find
  18. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of the 1st International Conference on Learning Representations. find
  19. Piero Molino, Yang Wang, and Jiawei Zhang. 2019. Parallax: Visualizing and understanding the semantics of embedding spaces via algebraic formulae. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 165–180, Florence, Italy. Associ… find
  20. Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1), pages 1896–1906, Melbourne, Australia. Association for Computational Li… find
  21. Apoorv Nandan. 2020. Text generation with a miniature gpt. https://github.com/keras-team/keras-io/blob/ master/examples/generative/text_ generation_with_miniature_gpt.py. ↗
  22. Michael P. Oakes. 1998. Statistics for Corpus Linguistics. Edinburgh University Press. find
  23. Karl Pearson. 1901. Liii. on lines and planes of closest fit to systems of points in space. Philosophical Magazine Series 1, 2:559–572. find
  24. Bénédicte Pincemin. 2024. Specificities and other applications of the Fisher’s exact test to textual data: What’s the matter with lexical frequencies? In JADT 2024 : Mots comptés, textes déchiffrés, volume 2 of Cahiers du Cental, pages 703–712, Bruxelles, Belgium. Presses universitaires de Louvain. find
  25. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language… find our card
  26. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. find
  27. Leslie Redmond, Denis Foucambert, and Lucie Libersan. 2023. Language Corpora and Principal Components Analysis, pages 117–132. Springer International Publishing, Cham. find
  28. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA. Association for Co… find our card
  29. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842– 866. find our card
  30. Matteo Saponati, Pascal Josef Sager, Pau Vil-imelis Aceituno, Thilo Stadelmann, and Benjamin F Grewe. 2025. The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in transformer training. In Proceedings of the 42nd International Conference on Machine Learning, v… find
  31. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626. find our card
  32. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics. find our card
  33. K. Simonyan, A. Vedaldi, and A. Zisserman. 2014. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014 - Workshop Track. find our card
  34. John Sinclair. 1991. Corpus, concordance, collocation. find
  35. Jure Skubic and Darja Fišer. 2024. Parliamentary discourse research in political science: Literature review. In Proceedings of the IV Workshop on Creating, Analysing, and Increasing Accessibility of Parliamentary Corpora (ParlaCLARIN) @ LREC-COLING 2024, pages 1–11, Torino, Italia. ELRA and ICCL. find
  36. Arman Engin Sucu, Yixiang Zhou, Mario A. Nascimento, and Tony Mullen. 2025. Exploiting contextual information to improve stance detection in informal political discourse with LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research… find
  37. Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng. 2015. Learning word representations by jointly modeling syntagmatic and paradigmatic relations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find
  38. Elena Tognini-Bonelli. 2002. Corpus linguistics at work. Computational Linguistics, 28:583–583. find
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Cur… find
  40. Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42, Florence, Italy. Association for Computational Linguistics. find our card
  41. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vanni-2026-explore-political-discourse-transformers-emergent/