📄 MADAD: A Readability Annotation Tool for Arabic Text

Summary

Arabic readability research has lacked large datasets in which human judges assign difficulty labels to texts, so there was no gold standard for testing automatic readability predictors. The authors present MADAD, a browser-based, collaborative annotation environment with an Arabic/English interface, built mainly to gather such judgments. Readability can be rated at the sentence, phrase or paragraph level in two ways: annotators either compare two texts or place a single text on a difficulty scale. MADAD also works as a general annotation tool. Its label inventories are not hard-coded; a manager uploads a user-defined XML Schema, so the tool can serve established and emerging NLP tasks alike. The system includes role-based task management, inter-annotator agreement scoring, adjudication and XML export. The paper describes the design only and reports no empirical evaluation; testing against other tools is left for future work. Its value lies in giving Arabic NLP a way to build readability gold standards, which formula-based measures such as Flesch cannot provide.

Key points

• The architecture has three layers: Services (corpus storage and annotation services), Executive (authentication and role assignment) and User interface. There are three roles: Administrator, Annotation Manager and Annotator.
• There are two readability methods. The first is pairwise comparison using manager-defined comparative statements; after each judgment, a new random pair of texts is shown. The second is direct rating on a scale whose default runs from 0 (easy) to 100 (difficult).
• Preprocessing offers two kinds of segmentation: sentences are split at punctuation such as full stops and semicolons, and words are split at whitespace.
• Agreement is measured with Cohen Kappa for two annotators and Fleiss Kappa for more. A visual comparison view and an adjudication function let the manager settle conflicts and produce the gold corpus.
• Results are exported as XML with token IDs, annotation level, type and features written as attribute=value. Planned evaluation follows the tool-evaluation criteria of Dipper et al. (2004).

Interesting ideas

• I1. Asking people which of two texts is harder is easier and more consistent than asking for an absolute grade, and the system stores each pair with the chosen statement.
• I2. The label inventory lives in an uploaded schema (elements, permitted attributes, allowed values) instead of in code, so new tasks need no new software.
• I3. In free-span annotation, agreement is computed only over tokens that both annotators actually labelled, which keeps differences in coverage from distorting the score.
• I4. Scoring agreement first and adjudicating afterwards, with a per-type visual diff, makes gold-standard construction a step that can be audited.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when the LLM teacher checks silver data, ask it to pick the better of two competing parses or frame readings rather than score each in isolation.
• I2 (frames): keep frame and semantic-role inventories as declarative, validated schema files, so adding a VerbNet-derived role set means editing data while every assigned label stays traceable to its definition.
• I3 (learn): when witness parsers disagree on which spans they analyse, compute agreement on the shared spans and report coverage separately, which gives a cleaner signal for accepting silver data.
• I4 (tree): show witness-parser disagreements as a per-relation diff and record each adjudication decision, so every gold arc keeps an explanation of why it won.