Elamur Research⌕
☰
home / papers / 2026 / ByteBreaker@DravidianLangTech 2026: XLM-RoBERTa Large with Sliding-Window Chunking and Top-K Mean Pooling for Writing Style Classification

ByteBreaker@DravidianLangTech 2026: XLM-RoBERTa Large with Sliding-Window Chunking and Top-K Mean Pooling for Writing Style Classification

Paper · 2026 · Chava Srinivasa Sai, R Vinay Kumar, Jigeesha Sai Surapaneni et al.

Summary

ByteBreaker was submitted to the DravidianLangTech@ACL 2026 shared task on recovering the prompts given to LLMs. The task is to infer which of nine target styles (such as Formal, Humorous, Pessimistic or Serious) an LLM was asked to apply when rewriting a Telugu document. Many texts exceed the 512-token input window, so the authors split each rewritten text into overlapping 512-token segments (stride 256). Each segment takes its document's label, and XLM-RoBERTa Large is fine-tuned on them. At prediction time, the document label comes from averaging only the three segments with the most confident logits (Top-K Mean Pooling). On validation, this beat plain averaging and LogSumExp. Models were trained with five seeds. The submissions were a grid-searched weighted ensemble and two single checkpoints. A single Top-K-guided checkpoint gave the best macro F1 (0.3306), while the ensemble gave the highest accuracy (0.3256). The results show that fine-grained style recovery in Telugu is still difficult. They also suggest that checkpoint selection should match the inference-time pooling, especially for rare classes.

Key points

  • The data come from the shared task (B et al., 2026), with 3,000 train, 300 validation and 301 test items. Only the LLM-rewritten "CHANGE STYLE" text was used for training, not the original transcript. In the test set, Serious is the largest class (15.61%) and Pessimistic the smallest (7.31%).
  • The model is xlm-roberta-large (24 layers, ≈560M parameters) with a linear head over [CLS]. Training used learning rate 1e-5, effective batch size 16, label smoothing 0.10, BF16, and early stopping on document-level macro F1. The seeds were {42, 7, 123, 2025, 99}.
  • Official macro F1 scores were 0.3290 for Run 1 (ensemble of four checkpoints), 0.3142 for Run 2 (mean-guided model) and 0.3306 for Run 3 (Top-K-guided model).
  • Optimistic and Humorous were the easiest classes (F1 ≈0.39–0.47). Serious was the hardest (F1 ≈0.10–0.14) and was often confused with Formal and Authoritative.
  • Adding the mean-guided model to the ensemble lowered Pessimistic precision from 0.50 to 0.30. Comparisons with Longformer and BigBird are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

Cite

Cite as Chava Srinivasa Sai, R Vinay Kumar, Jigeesha Sai Surapaneni and Chava Shanmukha Sai. 2026. ByteBreaker@DravidianLangTech 2026: XLM-RoBERTa Large with Sliding-Window Chunking and Top-K Mean Pooling for Writing Style Classification. Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.dravidianlangtech-1.19

@inproceedings{sai2026bytebreaker,
  title = {{ByteBreaker@DravidianLangTech 2026: XLM-RoBERTa Large with Sliding-Window Chunking and Top-K Mean Pooling for Writing Style Classification}},
  author = {Chava Srinivasa Sai, R Vinay Kumar, Jigeesha Sai Surapaneni and Chava Shanmukha Sai},
  booktitle = {Proceedings of the Sixth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages},
  year = {2026},
  doi = {10.18653/v1/2026.dravidianlangtech-1.19},
  url = {https://doi.org/10.18653/v1/2026.dravidianlangtech-1.19}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shlomo Argamon, Casey Whitelaw, Paul Chase, Sobhan Raj Hota, Navendu Garg, and Shlomo Levitan. 2007. Stylistic text classification using functional lexical features. Journal of the American Society for Information Science and Technology, 58(6):802–822. find
  2. Premjith B, Jyothish Lal G, Bharathi Raja Chakravarthi, Saranya Rajiakodi, Durairaj Thenmozhi, Ratnavel Rajalakshmi Rajalakshmi, Rahul Ponnusamy, and Chinthala Bhuvanesh. 2026. Shared task on prompt recovery for llm in telugu. In Proceedings of the Sixth Workshop on Speech, Vision, and Language Tech… find
  3. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020. Corpus creation for sentiment analysis in code-mixed tamil-english text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collabor…
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  6. Shaik Fharook, Syed Sufyan Ahmed, Gurram Rithika, Sumith Sai Budde, Sunil Saumya, and Shankar Biradar. 2022. Are you a hero or a villain? A semantic role labelling approach for detecting harmful memes. In Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emer… find
  7. Adeep Hande, Ruba Priyadharshini, and Bharathi Raja Chakravarthi. 2020. KanCMD: Kannada codemixed dataset for sentiment analysis and offensive language detection. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 54… find
  8. Sanjana M Kavatagi, Rashmi R Rachh, and Shankar S Biradar. 2023. VTUBGM@LT-EDI-2023: Hope speech identification using layered differential training of ULMFiT. In Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion, pages 209–213. find
  9. Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the American Society for Information Science and Technology, 60(1):9–26. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. et al. 2020. Transformers: State-of-the-art natural language process…
  11. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
  12. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big Bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283–17297.
  13. B Premjith, Bharathi Raja, and Prasanna Kumar Kumaresan. 2026. Findings of the shared task on prompt recovery for LLM in Telugu – Dravidian-LangTech@ACL 2026. In Proceedings of the Sixth Workshop on Speech and Language Technologies for Dravidian Languages (DravidianLangTech-2026), San Diego, Califor… find
  14. Chava Sai, Rangoori Kumar, Sunil Saumya, and Shankar Biradar. 2024. IIITDWD_SVC@DravidianLangTech-2024: Breaking language barriers; hate speech detection in Telugu-English code-mixed text. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pag… find
  15. Sunil Saumya, Vanshita Jha, and Shankar Biradar. 2022. Sentiment and homophobia detection on YouTube using ensemble machine learning techniques. In Working Notes of FIRE 2022 – Forum for Information Retrieval Evaluation (Hybrid). CEUR. find
  16. Prasha Shrestha, Sebastian Sierra, Fabio Gonzalez, Manuel Montes, Paolo Rosso, and Thamar Solorio. 2017. Convolutional neural networks for authorship attribution of short texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2:… find
  17. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification? In Proceedings of the 18th Chinese National Conference on Computational Linguistics (CCL), pages 194–206.
  18. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sai-2026-bytebreaker-dravidianlangtech-2026-xlm-roberta/