Elamur Research⌕
☰
home / papers / 2020 / EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian

EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian

Paper · 2020 · Valerio Basile, Danilo Croce, Maria Di Maro et al.

Summary

This paper is the organizers' overview of EVALITA 2020, the seventh edition of the biennial shared-task campaign for Italian language and speech technology. Because of COVID-19, the final workshop was held online. Across five tracks, the campaign ran 14 peer-reviewed tasks:

  • Creativity and Style
  • Semantics and Multimodality
  • Affect, Hate, and Stance
  • Time and Diachrony
  • New Challenges in Long-standing Tasks

Task organizers built the datasets, and the authors plan to release them on GitHub where source licenses allow. The overview describes each task briefly, analyzes participation by task, track, country and gender, and explains how the cross-task best-system award was judged.

Participation reached a record level, with 51 teams and 58 system papers. Systems based on Transformers, BERT in particular, won at least one subtask in most tasks. Tasks that reduce to plain classification or regression drew the most teams, while the generation task got no submissions.

The paper matters as a snapshot of Italian NLP benchmarking around 2020. It also gives practical lessons for anyone running evaluation campaigns for a mid-resource language.

Key points

  • Scale: 51 teams (about 130 people) submitted 240 runs and 58 system descriptions, roughly 70% more papers than EVALITA 2018. More than 180 authors came from 18 countries, 64% of them from Italy. The 59 task organizers came from 6 countries.
  • Uneven participation: HaSpeeDe had the most teams (14), followed by SardiStance (12) and DIACR-Ita (9). CHANGE-IT, a headline style transfer task and the first Italian NLG task, had zero participants. Ghigliottin-AI and other harder tasks attracted only a few teams.
  • New tasks: this edition introduced DANKMEMES (the first multimodal task, on memes about the 2019 Italian Government Crisis), DIACR-Ita (semantic shift), KIPoS (POS tagging on the KIParla spoken corpus), PRELEARN, CONcreTEXT and DaDoEval. No task used actual speech.
  • Dominant methods: in at least 11 of the 14 tasks, the best result in at least one subtask came from a neural architecture built on or combined with Transformers.
  • Award: organizers of 7 tasks nominated 10 systems for the best-system award. Of the 20 candidate authors, 12 were master's or PhD students.

Interesting ideas

  • I1. Drop the constrained/unconstrained split. Organizers were discouraged from separating runs this way, because pre-trained embeddings and language models blur the line between "only provided data" and "external resources." Benchmark designers for Ukrainian could rethink similar categories for the same reason.
  • I2. Schedule tasks in non-overlapping windows. Each task had its own evaluation period, so teams could enter several tasks without deadline clashes. The authors credit this choice with helping raise participation, and any multi-task campaign could adopt it.
  • I3. Evaluate through an API. Ghigliottin-AI used a Remote Evaluation Server, where systems answered over an API instead of uploading prediction files. This makes interactive or game-like evaluation possible and keeps the test data hidden.
  • I4. Judge awards on more than the leaderboard. Nominees did not need to rank first. They were assessed on novelty, linguistically motivated features or resources, depth of error analysis and discussion of limits, and methodological rigor. This rewards insight and encourages student work.
  • I5. Lower the barrier for complex tasks. The empty generation task and the sparse Ghigliottin-AI entries suggest that hard tasks need starter resources or baselines. Without them, participants gravitate toward tasks that can be framed as classification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 685 works

Cite

Cite as Valerio Basile, Danilo Croce, Maria Di Maro and Lucia C. Passaro. 2020. EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{basile2020evalita,
  title = {{EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian}},
  author = {Valerio Basile, Danilo Croce, Maria {Di Maro} and Lucia C. Passaro},
  booktitle = {Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)},
  year = {2020},
  doi = {10.4000/books.aaccademia.6747},
  url = {https://doi.org/10.4000/books.aaccademia.6747}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chiara Alzetta, Alessio Miaschi, Felice Dell’Orletta, Frosina Koceva, and Ilaria Torre. 2020. PRE-LEARN@EVALITA2020: Overview of the prerequisite relation learning task for italian. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campa… find
  2. Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020a. DIACR-Ita@EVALITA2020: Overview of the evalita2020 diachronic lexical semantics (diacr-ita) task. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th… find
  3. Pierpaolo Basile, Marco Lovetere, Johanna Monti, Antonioa Pascucci, Federico Sangati, and Lucia Siciliani. 2020b. Ghigliottin-AI@EVALITA2020: Evaluating artificial players for the language game “la ghigliottina”. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedi… find
  4. Cristina Bosco, Silvia Ballarè, Massimo Cerruti, Eugenio Goria, and Caterina Mauri. 2020. KIPoS@EVALITA2020:overview of the task on kiparla part of speech tagging. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural La… find
  5. Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta, Simonetta Montemagni, Giulia Venturi, and Roberto Zamparelli. 2020. AcCompl-it@EVALITA2020: Overview of the acceptability & complexity evaluation task for italian. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors,… find
  6. T. Caselli, N. Novielli, V. Patti, and P. Rosso. 2018. find
  7. EVALITA 2018: Overview of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. In Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso, editors, Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  8. Final Workshop (EVALITA 2018) , Turin, Italy. CEUR.org. find
  9. Alessandra Teresa Cignarella, Mirko Lai, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020. SardiStance@EVALITA2020: Overview of the task on stance detection in italian tweets. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaig… find
  10. Andrea Cimino, Felice Dell’Orletta, and Malvina Nissim. 2020. TAG-it@EVALITA2020: Overview of the topic, age, and gender prediction task for italian. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Language Processi… find
  11. Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Albert Gatt. 2020a. CHANGE-IT@EVALITA2020: Change headlines, adapt news, generate. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Languag… find
  12. Lorenzo De Mattei, Graziella De Martino, Andrea Iovine, Alessio Miaschi, Marco Polignano, and Giulia Rambelli. 2020b. ATE ABSITA@EVALITA2020: Overview of the aspect term extraction and aspect-based sentiment analysis task. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors… find
  13. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2020. AMI@EVALITA2020: Automatic misogyny identification. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA… find
  14. Lorenzo Gregori, Maria Montefinese, Daniele P. Radicioni, Andrea Amelio Ravelli, and Rossella Varvara. 2020. CONCRETEXT@EVALITA2020: The concreteness in context task. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural… find
  15. Stefano Menini, Giovanni Moretti, Rachele Sprugnoli, and Sara Tonelli. 2020. DaDoEval@EVALITA2020: Same-genre and cross-genre dating of historical documents. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural Language… find
  16. Martina Miliani, Giulia Giorgi, Ilir Rama, Guido Anselmi, and Gianluca E. Lebani. 2020. DANKMEMES@EVALITA2020: The memeing of life: memes, multimodality and politics. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, Proceedings of the 7th evaluation campaign of Natural… find
  17. Manuela Sanguinetti, Gloria Comandini, Elisa Di Nuovo, Simona Frenda, Marco Stranisci, Cristina Bosco, Tommaso Caselli, Viviana Patti, and Irene Russo. 2020. HaSpeeDe 2@EVALITA2020: Overview of the evalita 2020 hate speech detection task. In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C.… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/basile-2020-evalita-2020-overview-7th-evaluation/