Elamur Research⌕
☰
home / papers / 2026 / Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias

Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias

Paper · 2026 · Elias Schuhmacher, Andrianos Michail, Juri Opitz et al.

Summary

Embedding-based search can only surface information that a document's vector actually encodes. This paper asks whether every part of a long, multi-segment document is represented equally. The authors build a permutation-based diagnostic on a comparable Wikipedia corpus in six languages. Topic-aligned articles are concatenated in all possible orders, which separates the effect of position from the effect of content. Fairness is measured as the cosine similarity between the full-document embedding and each segment's standalone embedding, and position effects are estimated with OLS regression.

They test two multilingual long-context encoders, mGTE and jina-v3. Both models favour content placed first in the document. Their language preferences differ by model, and segments in higher-resource languages such as English tend to be favoured. Attention analysis of mGTE's <s> pooling token links the first-position bias to attention that is concentrated on early tokens. The authors therefore propose a training-free calibration applied at inference time, which spreads attention more evenly across the document. This makes later segments more discoverable without degrading semantics. The work matters for page-level retrieval, such as historical newspapers in which independent articles share one page.

Key points

  • Corpus construction:
    • Wikipedia articles in English, Chinese, German, Italian, Korean and Hindi, with each segment 1,000–2,000 tokens long.
    • Non-English articles were kept only if their length was within ±70% of the English article, counted in XLM-R subtokens.
    • Documents contain n ∈ {3, 4, 5, 6} segments, up to 8,192 tokens.
  • Dataset size: 1,002 documents for n=3, 1,008 for n=4, 1,080 for n=5, and 10,080 for n=6. Both monolingual and mixed-language configurations were used; in the mixed ones, the lead segment's language differs from the language of the later segments.
  • Models:
    • mGTE (Alibaba-NLP/gte-multilingual-base) pools via the <s> token.
    • jina-v3 (jinaai/jina-embeddings-v3) uses mean pooling with the text-matching LoRA adapter. For this model, information retention is also measured via late-chunking contextualized segment embeddings.
  • Statistical test: Positional fairness is tested by OLS with position indicators and standard errors clustered by segment set. The null hypothesis is that all position coefficients β_p equal zero.
  • Calibration and release:
    • The calibration was evaluated mainly on mGTE.
    • A control experiment suggests that it adds information about later segments rather than collapsing meaning.
    • The code is released at github.com/impresso/fair-sentence-transformers, within the Impresso project funded by SNSF and FNR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Elias Schuhmacher, Andrianos Michail, Juri Opitz, Rico Sennrich and Simon Clematide. 2026. Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.246

@inproceedings{schuhmacher2026information,
  title = {{Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias}},
  author = {Elias Schuhmacher, Andrianos Michail, Juri Opitz, Rico Sennrich and Simon Clematide},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.246},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.246}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hongliu Cao. 2025. Writing style matters: An examination of bias and fairness in information retrieval systems. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM ’25, page 336–344, New York, NY, USA. Association for Computing Machinery. find
  2. João Coelho, Bruno Martins, Joao Magalhaes, Jamie Callan, and Chenyan Xiong. 2024. Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 370–377,… find
  3. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solom… — MMTEB: Massive Multilingual Text Embedding Benchmark
  4. Mohsen Fayyaz, Ali Modarressi, Hinrich Schuetze, and Nanyun Peng. 2025. Collapse of dense retrievers: Short, early, and literal biases outranking factual evidence. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9136–9152, Vie… find
  5. A. Gelman, J. Hill, and A. Vehtari. 2021. Regression and Other Stories. Analytical Methods for Social Research. Cambridge University Press. find
  6. Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. 2025. Late chunking: Contextual chunk embeddings using long-context embedding models. Preprint, arXiv:2409.04701. — [arXiv:2409.04701](https://arxiv.org/abs/2409.04701) find
  7. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Reagan J. Lee, Samarth Goel, and Kannan Ramchandran. 2025. Quantifying positional biases in text embedding models. Preprint, arXiv:2412.15241. — [arXiv:2412.15241](https://arxiv.org/abs/2412.15241) find
  8. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173. Place: Cambridge, MA Publisher: MIT Press.
  9. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. 2025. Nolima: Long-context evaluation beyond literal matching. In Forty-second International Conference on Machine Learning. find
  10. Amirkeivan Mohtashami and Martin Jaggi. 2023. Random-access infinite context length for transformers. In Advances in Neural Information Processing Systems, volume 36, pages 54567–54585. Curran Associates, Inc. find
  11. Saahil Ognawala and Alex Cureton-Griffiths. 2025. Long-Context Embedding Models are Blind Beyond 4K Tokens. find
  12. Juri Opitz, Lucas Moeller, Andrianos Michail, Sebastian Padó, and Simon Clematide. 2025. Interpretable text embeddings and text similarity explanation: A survey. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22303–22319, Suzhou, China. Association f… find
  13. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. jina-embeddings-v3: Multilingual Embeddings With Task LoRA. arXiv preprint. ArXiv:2409.10173 [cs].
  14. Ziyang Zeng, Dun Zhang, Jiacheng Li, Zoupanxiang, Yudong Zhou, and Yuqing Yang. 2025. An empirical study of position bias in modern information retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 5069–5081, Suzhou, China. Association for Computational Linguisti… find
  15. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval. In Proceedings of the 2024…
  16. Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. LongEmbed: Extending embedding models for long context retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 802–816, Miami, Florida, USA. Association for Comp…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schuhmacher-2026-information-representation-fairness-long-document/