📄 SwissText 2024 Shared Task: Automatic Classification of the United Nations' Sustainable Development Goals (SDGs) and Their Targets in English Scientific Abstracts

Summary

This document describes the SwissText 2024 shared task, organised at the University of Zurich within the DIZH-funded "SDG Research Scout" project. The task links English scientific abstracts to the United Nations' Sustainable Development Goals. Task 1 asks systems to pick the single best SDG out of 17, or a separate non-relevant class. Training data are at least 400 labelled, unbalanced abstracts from the UZH repository Zora. Task 2 goes one level deeper: given an abstract and its SDG, systems predict a primary target within that goal, plus optional additional targets or an "unspecific target" label. Task 2 has no training data, only a 50-abstract development set, so organisers point participants toward zero-shot and few-shot prompting. Scoring uses accuracy and per-SDG averaged F1 for Task 1, and F1 for Task 2. The document reports no scores. It lists four system papers presented at a workshop, two of which build synthetic training data. The task matters because it offers a benchmark for tracking how academic research connects to global sustainability goals.

Key points

• Task 1 is single-label classification into 17 SDGs plus a "non-relevant" class. Its training set has at least 400 unbalanced abstracts from Zora (zora.uzh.ch).
• Task 1 metrics are accuracy (primary) and average F1 per SDG (secondary). The secondary metric rewards systems that handle rarely seen goals well.
• Task 2 is multi-label target prediction conditioned on a given SDG. It has no training data and a 50-abstract development set, and is scored with F1 against a manually corrected ground truth.
• External resources such as the OSDG-CD dataset are allowed but must be declared. Manual correction of test output is forbidden, and each test phase lasts 2 days.
• Four system papers were presented. Two contributions (MODUL University Vienna/webLyzard and Chur) focus on synthetic training data, one of them using explainable AI to probe class borders.

Interesting ideas

• I1. Reporting plain accuracy alongside per-class averaged F1 shows whether a system ignores rare classes, which a single overall number can hide.
• I2. Every label set includes an explicit escape option ("non-relevant", "unspecific target"), so systems are not forced to make a confident wrong choice.
• I3. Fine labels are predicted only inside an already chosen coarse label, and choosing the "unspecific" fine label rules out any extra fine labels.
• I4. When almost no labelled data exists, a small dev set is used only for tuning prompts or few-shot setups rather than for training.
• I5. One contribution uses explainable AI to find where class boundaries lie and generates synthetic examples there.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation of parsers and frame labellers could always pair overall scores with macro-averaged per-label scores, so rare relations or roles that silently fail become visible.
• I2 (answer): Readers answering tests or fable questions could have a principled "no supported answer" outcome when the text world lacks evidence, instead of guessing.
• I3 (world): Ontology lookups could work the same way, first fixing a coarse WordNet/FrameNet class and then refining within it, with an explicit "generic" node that blocks further specialisation.
• I4 (learn): LLM teacher tiers could be calibrated on a tiny hand-checked dev set before producing silver data, keeping that set clean as a check rather than as training material.
• I5 (learn): Self-play could aim generated sentences at constructions where witness parsers disagree, producing training data exactly at the parser's decision borders.