📄 Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance

Summary

Greek financial text has been largely neglected by LLM research. Greek has rich morphology and few domain resources, and no benchmark has targeted Greek finance specifically. The authors fill this gap with Plutus-ben, an evaluation suite of six tasks: recognizing named entities in both numeric and textual forms, answering questions, producing summaries both extractively and abstractively, and classifying topics. Four datasets were built from scratch from Greek annual reports and university exam material, with labels from native-speaker experts. Two earlier resources were added to complete the suite. The authors also fine-tuned Plutus-8B on Greek financial data and compared it with 24 LLMs. These ranged from English-centric to Greek models, general to financial, and small to very large. Plutus-8B obtained the best average score. Large commercial models were weaker than expected, and small open models broke down on NER. Neither English financial tuning nor general Greek training was enough, and larger scale gave little extra benefit. Summarizing long documents stayed difficult for every system. The work shows that in-language, domain-specific adaptation matters for low-resource specialized settings.

Key points

• Datasets:
  • New: GRFinNUM, GRFinNER, GRFinQA and GRFinSUM.
  • Reused: GRFNS-2023 (abstractive summarization) and GRMultiFin (topic classification).
  • GRFinNUM comes from 64 annual reports of Athens Stock Exchange firms, filtered to 500 sentences. Its numeric labels are MONETARY, PERCENTAGE, TEMPORAL, QUANTITY and OTHERS.
• Annotation quality (GRFinNUM): three expert annotators reached very high agreement, with F1 0.988, Cohen's Kappa 0.979 and Krippendorff's Alpha 0.978.
• Plutus-8B results:
  • It reached a mean score of 0.52 and 0.70 on GRFinNUM.
  • On the held-out GRFinQA and GRFinSUM it scored 0.64 and 0.17.
  • Meltemi-7B scored 0.48 and 0.09 on these, and Llama-Krikri-8B scored 0.57 and 0.05.
  • The reported margins over other models are 8.33% over GPT-4 and 26.83% over GPT-4o.
• Scaling: Qwen2.5-72B performed no better than its 32B counterpart.
• Human evaluation on GRFNS-2023:
  • GPT-4 led on fluency (4.97).
  • Plutus-8B scored 3.51 on coherence and 2.93 on factuality, close to GPT-4's 3.06 on factuality.
  • Meltemi-7B was fluent (3.99) but scored low on coherence (1.49).