Elamur Research⌕
☰
home / papers / 2026 / Flipper: An Extended Document-Level Financial Dataset for Training and Evaluation with Annotated Discourse Phenomena

Flipper: An Extended Document-Level Financial Dataset for Training and Evaluation with Annotated Discourse Phenomena

Paper · 2026 · Mariam Nakhlé, Rachel Atherly, Gabriela nicole Gonzalez Saez et al.

Summary

Machine translation still struggles with phenomena that depend on context outside the current sentence, such as pronouns, agreement and consistent terminology. These errors are costly in finance, where translated documentation is legally required. Document-level financial corpora are rare, and evaluation mostly relies on sentence-level metrics or artificial test suites. The authors build Flipper, a parallel corpus drawn from publicly available PDF documents of asset managers (annual reports, Key Information Documents, factsheets, ESG reports) in five language pairs. They extend the pipeline behind the earlier DOLFIN dataset (Nakhlé et al., 2025) in three ways: whole sections are aligned with a windowed LASER scoring scheme, near-duplicates are removed with MinHash after masking numbers, and existing tools plus custom scripts add inline XML tags for five context-sensitive phenomena. A phenomenon-dense test split is held out and the rest is used for training. Fine-tuning gemma-3-1b-it on Flipper improves COMET for three of five pairs and four of five phenomena. The resource supports both training and realistic, targeted document-level evaluation in a high-stakes domain.

Key points

  • Coverage and size: Flipper covers En–De, En–Es, En–Fr, En–It and Fr–Es. It contains about 122.9k parallel documents/sections, 3.6M sentences and roughly 118M tokens (source and target combined). It is released on Hugging Face as DragonLLM/Flipper.
  • Alignment and deduplication:
    • Each source sentence is scored by LASER against a target window, and the section score is the average of these scores.
    • The acceptance threshold of 0.80 was chosen after manually checking 1,200 sections.
    • Deduplication uses MinHash (text-dedup library) at threshold 0.5, with numbers masked so that templated fund documents collapse into one.
  • Annotated phenomena: five phenomena are tagged (the reordering of sentences, the form of verbs, consistent use of terms, gender and formality), giving 133,885 tags/documents overall.
    • Verb form is by far the most frequent (91,606, via MuDA).
    • Fr–Es received only reordering annotations, because the existing tools are English-centric.
  • Evaluation split: 2,000 sections per pair (10k in total), selected for the quantity and variety of annotation tags. This places it between a standard test set and a test suite.
  • Fine-tuning experiment: gemma-3-1b-it was fully fine-tuned with TRL for 1 epoch and scored with wmt22-comet-da.
    • Gains: En–Es 77.90→80.25, En–Fr 75.06→75.91, Fr–Es 78.19→78.87.
    • Drops: En–De 69.77→67.81, En–It 74.30→73.73.
    • By phenomenon, Formality (59.22→61.16) and Verb form (74.07→75.86) improve most, while Gender declines (77.33→75.62).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Université Grenoble AlpesSaint-Martin-d'Hères · France20 works · 8,059 citations🔬Laboratoire d'Informatique de GrenobleGrenoble · France21 works · 688 citations🎓Institut polytechnique de GrenobleGrenoble · France11 works · 207 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Discourse analysis · 466 works

Cite

Cite as Mariam Nakhlé, Rachel Atherly, Gabriela nicole Gonzalez Saez, Marco Dinarelli, Raheel Qader and Hervé Blanchon. 2026. Flipper: An Extended Document-Level Financial Dataset for Training and Evaluation with Annotated Discourse Phenomena. The 7th Financial Narrative Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/22f2djwbns96

@inproceedings{nakhle2026flipper,
  title = {{Flipper: An Extended Document-Level Financial Dataset for Training and Evaluation with Annotated Discourse Phenomena}},
  author = {Mariam Nakhlé, Rachel Atherly, Gabriela nicole Gonzalez Saez, Marco Dinarelli, Raheel Qader and Hervé Blanchon},
  booktitle = {The 7th Financial Narrative Processing Workshop},
  year = {2026},
  doi = {10.63317/22f2djwbns96},
  url = {https://doi.org/10.63317/22f2djwbns96}
}

References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks.
  2. Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. find
  3. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper…
  4. Gaëtan Caillaut, Raheel Qader, Jingshu Liu, Mariam Nakhlé, Arezki Sadoune, Massinissa Ah-mim, and Jean-Gabriel Barthelemy. 2025. The llm pro finance suite: Multilingual large language models for financial applications. 2020. A set of recommendations for assessing human–machine parity in language tra… find
  5. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Annual Conference of the European Association for Machine Translation, pages 261–268, Trento, Italy. European Association for Machine Translation.
  6. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923– 929, Portorož, Slovenia. European Language Resources Association (E…
  7. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural…
  8. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW ’18, page 1941–1942, Republic and Canton of G… find
  9. Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023. When does translation require context? a data-driven, multilingual exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 606–626, Toronto, C… find
  10. Pekka Malo, Ankur Sinha, Pyry Takala, Pekka Korhonen, and Jyrki Wallenius. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the American Society for Information Science and Technology. find
  11. Chenghao Mou, Chris Ha, Kenneth Enevoldsen, and Peiyuan Liu. 2023. Chenghaomou/text-dedup: Reference snapshot. find
  12. Simon Jehnen, Joaquín Ordieres-Meré, and Javier Villalba-Díez. 2025. Fintextsim: Enhancing financial text analysis with bertopic. find
  13. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. arXiv preprint arXiv:1810.02268.
  14. Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kento… — Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets
  15. Mariam Nakhlé, Marco Dinarelli, Raheel Qader, Emmanuelle Esperança-Rodier, and Hervé Blanchon. 2025. DOLFIN - document-level financial test-set for machine translation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5544–5556, Albuquerque, New Mexico. Association for… find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  17. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nakhle-2026-flipper-extended-document-level-financial/