Elamur Research⌕
☰
home / papers / 2019 / OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure

OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure

Paper · 2019 · Imad Zeroual, Dirk Goldhahn, Thomas Eckart et al.

Summary

Arabic is widely used online, yet freely available annotated Arabic corpora remain scarce. Many existing collections are raw, paid, or available only on request. The authors address this with OSIAN, an open corpus of international Arabic news. They took pages from 31 freely accessible broadcasters and used a customized version of the Leipzig Corpora Collection's CURL portal with the Heritrix crawler. The pages then went through the LCC language-independent pipeline: text extraction from WARC, sentence splitting, and regex-based removal of non-sentences. A custom tool split the crawl by web domain and mapped each article to a country. Every word was tagged with lemma and part of speech by a TreeTagger model adapted for Arabic, which was retrained with the Frequency Dictionary of Arabic. The result has about 3.5 million articles, over 37 million sentences and close to 1 billion tokens, stored as XML with article-level metadata. The corpus is archived in CLARIN, which makes it findable, searchable and reusable. That openness matters for Arabic NLP, information retrieval, and studies of variation across countries and over time.

Key points

  • Scale: roughly 6 million pages crawled in March 2018, reduced after cleaning to about 3.5M articles, 37M+ sentences and around 1B tokens. The sources were 31 international Arabic news platforms.
  • Tooling: an adapted CURL portal from the Leipzig Corpora Collection, Heritrix for crawling, and the LCC processing chain. A custom extractor splits articles by domain, and sub-corpora are formed per country.
  • Annotation: lemma and PoS come from an Arabic TreeTagger model. It was improved with Buckwalter & Parkinson's Frequency Dictionary of Arabic, which covers the top 5,000 words drawn from 30M words of corpora.
  • XML format: each article records source, date, URL, topic and language. It keeps the running text alongside a per-sentence, per-word layer holding surface form, PoS and lemma.
  • CLARIN distribution: metadata via OAI-PMH, plain text under CC BY-NC 4.0, and integration with WebLicht and Federated Content Search (FCS). Planned additions are KonText/Manatee and FCS-QL 2.0 querying of the annotation layers.

Interesting ideas

  • I1. The corpus is built from a curated list of trusted outlets rather than an unselective crawl, so its makeup is known before construction and less cleanup is needed afterwards.
  • I2. Each article is tied to its web domain and then to a country, which creates comparable sub-corpora that come almost for free.
  • I3. The tagger was strengthened by retraining it with a curated frequency dictionary rather than by adding more annotated sentences.
  • I4. The authors check the output by looking at frequency ranks of salient names and events, such as Syria and Trump ranking high, which shows the corpus reflects its period.
  • I5. Raw text, article metadata such as date and topic, and token-level annotation are stored together in one uniform record.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data for training could come from a named, curated set of sources, so that the composition of each batch can be audited and blamed when a witness parser fails.
  • I2 (learn): Splitting evaluation sets by source or genre would let regressions show up per slice instead of disappearing inside a single average score.
  • I3 (tokens): A compact list of high-frequency lemmas with their PoS could serve as a lexicon prior for the English tagger and lemmatizer, fixing frequent words cheaply.
  • I4 (learn): Frequency ranks of known entities in silver data can act as a quick sanity check before training, revealing skew from bad sources.
  • I5 (world): Keeping a publication date and source with every document gives the world builder an anchor for event ordering and lets each derived fact be traced back to its origin.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Leipzig UniversityLeipzig · Germany26 works · 8,622 citations🎓Mohamed I UniversityOujda · Morocco4 works · 433 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksLiterature and Literary Theory · 541 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 3,066 works

Cite

Cite as Imad Zeroual, Dirk Goldhahn, Thomas Eckart and Abdelhak Lakhouaja. 2019. OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure. Proceedings of the Fourth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-4619

@inproceedings{zeroual2019osian,
  title = {{OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure}},
  author = {Imad Zeroual, Dirk Goldhahn, Thomas Eckart and Abdelhak Lakhouaja},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Workshop},
  year = {2019},
  doi = {10.18653/v1/w19-4619},
  url = {https://doi.org/10.18653/v1/w19-4619}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tressy Arts, Yonatan Belinkov, Nizar Habash, Adam Kilgarriff, and Vit Suchomel. 2014. arTenTen: Arabic Corpus and Word Sketches. Journal of King Saud University - Computer and Information Sciences, 26(4):357–371. ↗
  2. Uwe Quasthoff, Dirk Goldhahn, and Thomas Eckart. 2014. Building large resources for text mining: The Leipzig Corpora Collection. In Text Mining, pages 3–24. Springer. ↗
  3. Motaz K. Saad and Wesam Ashour. 2010. Osac: Open source arabic corpora. In proceeding of the 6th International Conference on Electrical and Computer Systems (EECS’10), volume 10. ↗
  4. Tim Buckwalter and Dilworth Parkinson. 2014. A frequency dictionary of Arabic: Core vocabulary for learners. Routledge. ↗
  5. Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009. Clueweb09 data set. Available at http://lemurproject.org/clueweb09/. ↗
  6. Vít Suchomel and Jan Pomikálek. 2012. Efficient web crawling for large text corpora. In proceedings of the seventh Web as Corpus Workshop (WAC7), pages 39–43. find
  7. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017. ANT Corpus: An Arabic News Text Collection for Textual Classification. In proceedings of the 14th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA 2017), pages 135–142, Hammamet, Tunisia. find
  8. Reem Swuaileh, Mucahid Kutlu, Nihal Fathima, Tamer Elsayed, and Matthew Lease. 2016. ArabicWeb16: A New Crawl for Today’s Arabic Web. In proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 673–676. ACM. find
  9. Franciska de Jong, Bente Maegaard, Koenraad De Smedt, Darja Fišer, and Dieter Van Uytvanck. 2018. CLARIN: Towards FAIR and Responsible Data Science Using Language Resources. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 3259–3264. find
  10. Wajdi Zaghouani. 2017. Critical survey of the freely available Arabic corpora. arXiv preprint arXiv:1702.07835. ↗
  11. Imad Zeroual and Abdelhak Lakhouaja. 2018a. Arabic Corpus Linguistics: Major Progress, but Still a Long Way to Go. In Intelligent Natural Language Processing: Trends and Applications, Studies in Computational Intelligence, pages 613–636. Springer, Cham. ↗
  12. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages. In LREC, pages 759–765.
  13. Imad Zeroual and Abdelhak Lakhouaja. 2018b. Data science in light of natural language processing: An overview. Procedia Computer Science, 127:82–91. A Appendices Region or country International Middle-east Algeria Australia Canada China Egypt France Iran Iraq Germany Jordon Morocco Palestine Qatar Russia Saudi Arabia Sweden Syria Tunisia Turkey UAE UK USA Web-domain news.un.org arabic.euronews.com ara.reuters.com namnewsnetwork.org arabic.sputniknews.com aljazeera.net alarabiya.net djazairess.com eltelegraph.com arabnews24.ca halacanada.ca arabic.cctv.com alwatanalarabi.com france24.com alalam.ir iraqakhbar.com dw.com sarayanews.com www.marocpress.com al-ayyam.ps raya.com arabic.rt.com alwatan.com.sa alkompis.se syria.news www.turess.com turkey-post.net aa.com.tr emaratalyoum.com bbc.com arabic.cnn.com Nb. of articles 693,629 366,211 588,514 4,614 30,135 1,365 85,351 74,718 344,011 28,248 117,261 49,461 188,045 81,495 8,986 57,238 1,512 33,790 36542 495,674 76,638 25,081 10,686 113,557 *Table 1: List of crawled web-domains* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeroual-2019-osian-open-source-international-arabic/