Elamur Research⌕
☰
home / papers / 2014 / Multilingual Summarization Experiments on English, Arabic and French (Résumé Automatique Multilingue Expérimentations sur l'Anglais, l'Arabe et le Français) [in French]

Multilingual Summarization Experiments on English, Arabic and French (Résumé Automatique Multilingue Expérimentations sur l'Anglais, l'Arabe et le Français) [in French]

Paper · 2014 · Houda Oufaida, Omar Nouali and Philippe Blache

Summary

The paper tackles multi-document summarization that should transfer across languages with as little language-specific machinery as possible. The authors build a purely statistical extractive summarizer. It first groups sentences with hierarchical agglomerative clustering under cosine similarity. It then weights each term with an adaptation of mRMR (minimum Redundancy – Maximum Relevance):

  • Relevance is the mutual information between a term and the cluster label.
  • Redundancy is the term's average mutual information with all other terms.

A sentence's score is the mean mRMR weight of its terms. Sentences are then picked by a two-speed procedure (fast or slow) that fits short or very short summaries. On TAC MultiLing 2011 data in English, Arabic and French, they add basic processing layers one at a time: sentence splitting, tokenization, stop-word filtering and stemming. The system beats the baseline, and stop-word filtering gives the best results. ROUGE barely changes across layers, yet the selected sentences change a lot. The authors conclude that ROUGE alone hides real differences and that manual evaluation is needed.

Key points

  • Evaluation data is TAC MultiLing 2011, a parallel corpus covering 7 languages. Experiments use English, Arabic and French.
  • Term weights come from mRMR, combining relevance and redundancy either as a difference or as a quotient (Peng et al., 2005).
  • Four processing configurations are compared: SR, TSR, WSR and SSR, adding splitting, tokenization, stop-word removal and stemming in turn.
  • Sentence choice vs. ROUGE:
    • Between configurations, only about 40% of selected sentences agree on average, and never more than 50%.
    • Over 70% of the sentences chosen under SR are dropped under SSR.
    • ROUGE scores nonetheless stay nearly identical.
  • Stemmers and per-language effects:
    • The Arabic stemmer reduces words to 3–4 letter Semitic roots and causes the biggest disagreement.
    • The French Snowball stemmer makes only light changes and causes the most agreement.
    • In English, the long stop-word list drives the largest shift.
    • Arabic results are about 10% lower than the other languages.

Interesting ideas

  • I1. Weighting a term by how sharply its frequency differs across clusters of similar sentences, minus its overlap with other terms, rewards words that are both topical and non-redundant. Plain frequency or TF-IDF cannot make that distinction.
  • I2. Measuring how much the system's actual choices overlap between pipeline variants exposes large behavioural changes that an aggregate metric (here ROUGE) completely hides.
  • I3. Ablating preprocessing steps one at a time, and inspecting what each tool actually did (light Snowball vs. aggressive root extraction), shows that tool quality is a confounding variable. The effect should not be credited to the processing step itself.
  • I4. Questioning whether performance stability across languages should itself be an evaluation criterion is a useful framing for any system meant to generalise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): When deciding which entities or predicates are central to a text's world, a relevance-minus-redundancy score over event clusters could rank salient participants. It would avoid counting near-synonymous mentions twice.
  • I2 (learn): Alongside accuracy, evaluations should report the agreement rate of outputs (parses, frames, answers) between versions. That way a "same score, different behaviour" regression gets flagged.
  • I3 (tokens): When testing lemmatizer or morphology changes in crate en, diff the produced lemmas on a sample. This tells whether a downstream change comes from the idea or from an overly aggressive or weak implementation.
  • I4 (answer): Track variance of reader accuracy across genres (school tests, word problems, fables) as a metric in its own right. Do not rely only on the mean.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 5Research subjects and languages the work is about.English · 2,379 worksSemitic languages · 3,066 worksMultilingual & cross-lingual NLP · 882 worksSummarization · 243 worksFrench · 83 works

Cite

Cite as Houda Oufaida, Omar Nouali and Philippe Blache. 2014. Multilingual Summarization Experiments on English, Arabic and French (Résumé Automatique Multilingue Expérimentations sur l'Anglais, l'Arabe et le Français) [in French]. Proceedings of TALN 2014 (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{oufaida2014multilingual,
  title = {{Multilingual Summarization Experiments on English, Arabic and French (Résumé Automatique Multilingue Expérimentations sur l'Anglais, l'Arabe et le Français) [in French]}},
  author = {Houda Oufaida, Omar Nouali and Philippe Blache},
  booktitle = {Proceedings of TALN 2014 (Volume 2: Short Papers)},
  year = {2014},
  url = {https://aclanthology.org/F14-2026/}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B ARZILAY R., M C K EOWN K. R. (2005). Sentence Fusion for Multidocument News Summarization. Comput. Linguist., 31(3), 297–328.
  2. B OUDIN F., T ORRES -M ORENO J.-M. (2009). Résumé automatique multi-document et indépendance de la langue: une première évaluation en français. Actes de Traitement Automatique de La Langue Naturelle (TALN’09) find
  3. C ONROY J., D AVIS S. T., K UBINA J., L IU Y.-K., O’L EARY D. P., S CHLESINGER J. D. (2013). Multilingual Summarization: Dimensionality Reduction and a Step Towards Optimal Term Coverage. find
  4. Actes de MultiLing 2013 Workshop on Multilingual Multi-document Summarization, 55–63. find
  5. C ONROY J. M., S CHLESINGER J. D., K UBINA J., R ANKEL P. A., O’L EARY , D. P. (2011). CLASSY 2011 at TAC: Guided and multi-lingual summaries and evaluation metrics. Actes de Text Analysis Conference. find
  6. D AS P., S RIHARI R. (2011). Global and Local Models for Multi-Document Summarization. Actes de Text Analysis Conference. find
  7. D ING C., P ENG H. (2003). Minimum Redundancy Feature Selection from Microarray Gene Expression Data. Actes de IEEE Computer Society Conference on Bioinformatics (CSB'03), 523–529. find
  8. G IANNAKOPOULOS G., E L -H AJ M., F AVR , B., L ITVAK M., S TEINBERGER J., V ARMA V. (2011). TAC 2011 MultiLing Pilot Overview. Actes de Text Analysis Conference (TAC2011). find
  9. H MIDA F., F AVRE B. (2011). LIF at TAC Multiling: Towards a Truly Language Independent Summarizer. Actes de Text Analysis Conference (TAC). find
  10. L EDENEVA , Y., 2008. Effect of preprocessing on extractive summarization with maximal frequent sequences, Actes de MICAI 2008: Advances in Artificial Intelligence, 123–132. find
  11. L IN , C. (2004). Rouge: A Package for Automatic Evaluation of Summaries. Actes de Text Summarization Branches Out: ACL-04 Workshop, 74–81.
  12. L ITVAK M., L AST M., F RIEDMAN M. (2010). A New Approach to Improving Multilingual Summarization Using a Genetic Algorithm. Actes de 48th Annual Meeting of the Association for Computational Linguistics, 927–936. find
  13. M ARCU D. C. (1998). The Rhetorical Parsing, Summarization, and Generation of Natural Language Texts. Actes de 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics, 96-103. find
  14. M IHALCEA R., T ARAU P. (2005). A Language Independent Algorithm for Single and Multiple Document Summarization. Actes de International Joint Conference on Natural Language Processing (IJCNLP), Vol. 5. find
  15. P ENG H., L ONG F., D ING C. (2005). Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(8), 1226–1238. find
  16. S AGGION H. (2011). Using SUMMA for Language Independent Summarization at TAC 2011. Actes de Text Analysis Conference. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oufaida-2014-multilingual-summarization-experiments-english-arabic/