Elamur Research⌕
☰
home / papers / 2026 / Effect of Data Augmentation with Multi-View Perspectives of Signers on the DGS-Fabeln-1 Dataset

Effect of Data Augmentation with Multi-View Perspectives of Signers on the DGS-Fabeln-1 Dataset

Paper · 2026 · Fabian Renner, Daksitha Withanage Don, Elisabeth Andre et al.

Summary

The paper tackles German Sign Language (DGS) to German translation in a low-resource narrative setting, where data is scarce and most corpora record only a frontal view of the signer. The authors adapt the UniSign Transformer framework to the DGS-Fabeln-1 dataset, which has one native signer retelling seven fairy tales captured by several synchronized cameras. They extract skeletal pose keypoints with OpenPose and run three experiments:

  • comparing six versus seven fairy tales for training, to measure the effect of dataset size;
  • comparing random initialization with weights pretrained on multilingual data;
  • using only the frontal camera versus adding all available perspectives as data augmentation.

The test set stays fixed across all setups, so the results are directly comparable. More training material and the multi-view inputs both brought clear gains, and the best configuration reached a BLEU-4 of 4.20%. The work presents itself as the first competitive baseline for continuous translation on DGS-Fabeln-1. It also suggests that multi-angle recordings can help in domains with little data and creative language.

Key points

  • Dataset: DGS-Fabeln-1 (Nunnari et al., 2024) contains about 1 h 32 min of signing aligned with 1,428 German sentences. It has 573 video segments, each recorded from 8 views: a frontal camera, cameras at 30°, 60° and 90° on both sides, and an overhead camera capturing the interlocutor's back-channeling.
  • Data configurations (train/validation/test):
    • Six tales, frontal view only: 338/42/43.
    • Seven tales, frontal view only: 476/52/43.
    • Seven tales, all views: 3,749/409/43, with every view of a sentence kept in the same split.
  • Test set: "Snow White" was kept out of the test sets, and the test split is identical across all configurations.
  • Features and model: OpenPose extracts 134 keypoints covering hands, body and face. UniSign processes these with per-body-part GCN encoders followed by ST-GCN temporal encoders.
  • Result: Increasing the dataset size and adding multi-view augmentation significantly improved translation quality, with the top configuration scoring 4.20% BLEU-4.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AugsburgAugsburg · Germany8 works · 38,342 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 worksHuman-Computer Interaction · 141 works

Cite

Cite as Fabian Renner, Daksitha Withanage Don, Elisabeth Andre and Cristina Luna Jimenez. 2026. Effect of Data Augmentation with Multi-View Perspectives of Signers on the DGS-Fabeln-1 Dataset. Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2zrdptjrx3pa

@inproceedings{renner2026effect,
  title = {{Effect of Data Augmentation with Multi-View Perspectives of Signers on the DGS-Fabeln-1 Dataset}},
  author = {Fabian Renner, Daksitha Withanage Don, Elisabeth Andre and Cristina Luna Jimenez},
  booktitle = {Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion},
  year = {2026},
  doi = {10.63317/2zrdptjrx3pa},
  url = {https://doi.org/10.63317/2zrdptjrx3pa}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eleftherios Avramidis, Vera Czehmann, Fabian Deckert, Lorenz Hufe, Aljoscha Lipski, Yuni Amaloa Quintero Villalobos, Tae Kwon Rhee, Mengqian Shi, Lennart Stölting, Fabrizio Nunnari, and Sebastian Möller. 2025. The TUB Sign Language Corpus Collection. In Adjunct Proceedings of the 25th ACM Internatio… — [doi:10.1145/3742886.3756709](https://doi.org/10.1145/3742886.3756709) find
  2. Lucas Bernhard, Fabrizio Nunnari, Amelie Unger, Judith Bauerdiek, Christian Dold, Marcel Hauck, Alexander Stricker, Tobias Baur, Alexander Heimerl, Elisabeth André, Melissa Reinecker, Cristina España Bonet, Yasser Hamidullah, Stephan Busemann, Patrick Gebhard, Corinna Jäger, Sonja Wecker, Yvonne Kos… — [doi:10.1145/352919](https://doi.org/10.1145/352919) find
  3. Fabian Bross. 2020. Object marking in German Sign Language (Deutsche Gebärdensprache): Differential Object Marking and Object Shift in Necati Cihan Camgöz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. find
  4. Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10020–10030, Online. IEEE Computer Society. doi:10.1109/ CVPR42600.2020.01004. find
  5. Rina Damdoo and Praveen Kumar. 2025. SignEdgeLVM transformer model for enhanced Sign Language Translation on Edge Devices. Discover Computing, 28(1). doi:10.1007/s10791-025-09509-1. — [doi:10.1007/s10791-025-09509-1](https://doi.org/10.1007/s10791-025-09509-1) find
  6. Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. 2023. Machine Translation from Signed to Spoken Languages: State of the art and Challenges. Universal Access in the Information Society, 23(3):1305–1331. doi:10.1007/s10209-0 23-00992-1. — [doi:10.1007/s10209-0](https://doi.org/10.1007/s10209-0) find
  7. Amanda Cardoso Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giró i Nieto. 2021. How2Sign: A Large-Scale Multimodal Dataset for Continuous American Sign Language. In CVPR, pages 2735–2744, Online. Computer Vision Foundation / IEEE.… find
  8. Sergio Esteban-Romero, Iván Martín-Fernández, Manuel Gil-Martín, David Griol-Barres, Zoraida Callejas-Carrión, and Fernando Fernández-Martínez. 2024. LLM-Driven Multimodal Fusion for Human Perception Analysis. In Proceedings of the 5th on Multimodal Sentiment Analysis Challenge and Workshop: Social… — [doi:10.1145/3689062.3689084](https://doi.org/10.1145/3689062.3689084) find
  9. Sang-Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. 2019. Neural Sign Language Translation Based on Human Keypoint Estimation. Applied Sciences, 9(13):2683. doi:10.3 390/app9132683. find
  10. Reiner Konrad, Thomas Hanke, Gabriele Langer, Dolly Blanck, Julian Bleicken, Ilona Hofmann, Olga Jeziorski, Lutz König, Susanne König, Rie Nishio, Anja Regen, Uta Salden, Sven Wagner, Satu Worseck, Oliver Böse, Elena Jahn, and Marc Schulder. 2020. MEINE DGS – annotiert. Öffentliches Korpus der Deuts… — [doi:10.25592/dgs.corpus-3.0](https://doi.org/10.25592/dgs.corpus-3.0) find
  11. Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, and Houqiang Li. 2024. Uni-Sign: Toward Unified Sign Language Understanding at Scale. In The Thirteenth International Conference on Learning Representations, Singapore, Singapore. URL: https://iclr.cc/ virtual/2025/poster/31250. ↗
  12. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. URL: https://aclanthology.org/W04-1 013/. ↗
  13. Cristina Luna-Jiménez, Lennart Eing, Annalena Bea Aicher, Fabrizio Nunnari, and Elisabeth André. 2025. Lightweight Transformers for Isolated Sign Language Recognition. In Proceedings of the 27th International Conference on Multimodal Interaction, ICMI ’25, page 155–163, Canberra, Australia. Associat… — [doi:10.1145/3716553.3750772](https://doi.org/10.1145/3716553.3750772) find
  14. Cristina Luna-Jiménez, David Griol, and Zoraida Callejas. 2024. Zero-shot ensemble of language models for fine-grain mental-health topic classification. In Artificial Intelligence for Neuroscience and Emotional Systems: 10th International Work-Conference on the Interplay Between Natural and Artifici… — [doi:10.1007/978-3-031-61140-7_9](https://doi.org/10.1007/978-3-031-61140-7_9) find
  15. Claudia Macht and Markus Steinbach. 2019. 33. Regionalsprachliche Merkmale in der Deutschen Gebärdensprache, pages 914–935. De Gruyter Mouton, Berlin, Boston. URL: https://doi.org/10.1515/9783110261295-033 [cited 2026-01-26]. — [doi:10.1515/9783110261295-033](https://doi.org/10.1515/9783110261295-033) ↗
  16. Ozge Mercanoglu Sincan and Richard Bowden. 2025. Spotter+GPT: Turning Sign Spottings into Sentences with LLMs. In Adjunct Proceedings of the 25th ACM International Conference on Intelligent Virtual Agents, IVA Adjunct ’25, Berlin, Germany. Association for Computing Machinery. doi:10.1145/3742886.375… — [doi:10.1145/3742886.3756708](https://doi.org/10.1145/3742886.3756708) find
  17. Fabrizio Nunnari, Eleftherios Avramidis, Cristina España-Bonet, Marco González, Anna Hennes, and Patrick Gebhard. 2024. DGS-Fabeln-1: A Multi-Angle Parallel Corpus of Fairy Tales between German Sign Language and German Text. In Proceedings of the 2024 Joint International Conference on Computational… find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  19. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2025. A Survey of Multilingual Large Language Models. Patterns, 6(1):101118. doi: https://doi.org/10.1016/j.patter.2 024.101118. — [doi:10.1016/j.patter.2](https://doi.org/10.1016/j.patter.2) ↗
  20. Oline Ranum, David Wessels, Gomèr Otterspeer, Erik J Bekkers, Floris Roelofsen, and Jari I. Andersen. 2024. The NGT200 dataset - geometric multi-view isolated sign recognition. In ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling. URL: https://openreview.net/for… ↗
  21. Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró i Nieto. 2023. Sign language translation from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR): Workshops, pages 5625–5635, Vancouver, Canada. IEEE Computer S… — [doi:10.1109/cvprw59228.2023.00596](https://doi.org/10.1109/cvprw59228.2023.00596) find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Long Beach, California…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/renner-2026-effect-data-augmentation-multi-view/