Elamur Research⌕
☰
home / papers / 2016 / Creation of comparable corpora for English-Urdu, Arabic, Persian

Creation of comparable corpora for English-Urdu, Arabic, Persian

Paper · 2016 · Murad Abouammoh, Kashif Shah and Ahmet Aker

Summary

Statistical MT needs parallel text, which is scarce for many language pairs and domains. Most earlier work on comparable corpora targeted European languages, so this paper builds news-based comparable corpora pairing English with Arabic, Persian and Urdu. Articles were collected from hand-picked RSS feeds every 15 minutes and stored in weekly bins, with collection running from November 2014 toward May 2016. Documents are paired only across bins at most 7 days apart. For each English article, nouns are extracted with OpenNLP and scored by position, with the title and first sentence weighted most. The top 35% are translated using GIZA++ lexical dictionaries derived from OPUS alignments. These translations form a Lucene query, and the retrieved candidates are accepted if their cosine similarity is at least 0.1. The result is about 48.6K English-Arabic pairs, a few thousand pairs each for Persian and Urdu, and English-pivoted three-language tuples. Urdu had no previously known comparable corpus. The paper does not evaluate alignment quality or test the corpora in SMT. The data is available on request.

Key points

  • Sources: 289 Arabic, 270 English, 292 Persian and 239 Urdu RSS feeds, drawn from 44, 48, 23 and 16 websites respectively, covering domains such as politics, sport, economy and local news.
  • Monolingual totals:
    • Arabic: about 1.17M articles and 611M words over 60 weeks.
    • English: about 368K articles and 151M words over 60 weeks.
    • Persian: about 454K articles and 300M words over 41 weeks.
    • Urdu: about 133K articles and 56M words over 55 weeks.
  • Document alignments:
    • Bilingual: English-Arabic 48,646, English-Persian 4,716 and English-Urdu 3,434.
    • Trilingual: English-Urdu-Arabic 921, English-Urdu-Persian 682 and English-Arabic-Persian 1,017.
  • Core-term heuristic: the authors studied 1.7K articles averaging 206 user comments each.
    • Overall, 35% of article terms reappeared in the comments.
    • Title terms reappeared at 55% and first-sentence terms at 60%, compared with roughly 45% for sentences 2–6 and 33% for the rest.
  • Translation dictionaries were built with Moses scripts from GIZA++ alignments on OPUS data: 955K Persian-English, 10M Arabic-English and 726K Urdu-English sentence pairs. Only the single most probable translation of each term is used.

Interesting ideas

  • I1. Reader comments serve as an outside judge of which article terms really matter, and that evidence was used to set both the positional weights and the 35% cut-off.
  • I2. A very simple, fully explainable scoring scheme ranks nouns by section: 2 points each for appearing in the title or first sentence, 1 point each for the next five sentences or the remainder.
  • I3. Data is stored in weekly bins, and candidate matching is restricted to a 7-day window, which shrinks the search space and allows growth experiments by adding weeks one at a time.
  • I4. Retrieval works in two stages: a cheap Lucene top-10 recall step, then a stricter cosine threshold to accept a pair.
  • I5. English acts as a pivot so that bilingual links compose into three-language document tuples without direct alignment between the other languages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Mova could derive positional salience priors for entities from external evidence, so that the protagonist or topic of a text is ranked by justified weights rather than guesswork.
  • I2 (answer): A transparent score like this could pick the focal entities of a fable or school text when a question asks what the story is mainly about, with every point traceable.
  • I3 (learn): Silver data could be sliced into time or batch bins so that Mova can measure how quality changes as data grows and can check that gains are not artifacts of one batch.
  • I4 (learn): A fast candidate filter followed by an explicit, inspectable acceptance threshold could screen sentences before costly multi-parser agreement or LLM teacher review.
  • I5 (back): Translation through the meta-world could likewise link texts in several languages to one shared meaning graph, letting cross-language correspondences arise transitively.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom76 works · 20,808 citations🎓King Saud UniversityRiyadh · Saudi Arabia16 works · 293 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 English · 1,373 works Semitic languages · 1,484 works

Cite

Cite as Murad Abouammoh, Kashif Shah and Ahmet Aker. 2016. Creation of comparable corpora for English-Urdu, Arabic, Persian. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{abouammoh2016creation,
  title = {{Creation of comparable corpora for English-Urdu, Arabic, Persian}},
  author = {Murad Abouammoh, Kashif Shah and Ahmet Aker},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/5oqsboj3a5gm},
  url = {https://doi.org/10.63317/5oqsboj3a5gm}
}

References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Rauf, S., & Schwenk, H. (2009). Exploiting comparable corpora with TER and TERp. Paper presented at the Proceedings of the 2nd Workshop on Building and Using Comparable Corpora: from Parallel to Non-parallel Corpora. find
  2. Aker, A., Kanoulas, E., & Gaizauskas, R. J. (2012). A light way to collect comparable corpora from the Web. Paper presented at the LREC. find
  3. Barzilay, R., & McKeown, K. R. (2001). Extracting paraphrases from a parallel corpus. Paper presented at the Proceedings of the 39th Annual Meeting on Association for Computational Linguistics. our card
  4. Brown, P. F., Pietra, V. J. D., Pietra, S. A. D., & Mercer, R. L. (1993). The mathematics of statistical machine translation: parameter estimation. Comput. Linguist., 19(2), 263-311. find
  5. Callison-Burch, C., Koehn, P., & Osborne, M. (2006). Improved statistical machine translation using paraphrases. Paper presented at the Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. find
  6. DictMetric: ACCURAT D2.6 (2011). Toolkit for multi-level alignment and information extraction from comparable corpora, version 3.0. 29th June, 2012 164 pages. (http://www.accurat-project.eu/), ↗
  7. Hashemi, H. B., & Shakery, A. (2014). Mining a Persian– English comparable corpus for cross-language information retrieval. Information Processing & Management, 50(2), 384-398. ↗
  8. Hashemi, H. B., Shakery, A., & Faili, H. (2010). Creating a Persian-English comparable corpus Multilingual and Multimodal Information Access Evaluation (pp. 27-39): Springer. find
  9. Huang, D., Zhao, L., Li, L., & Yu, H. (2010). Mining large-scale comparable corpora from Chinese-English news collections. Paper presented at the Proceedings of the 23rd International Conference on Computational Linguistics: Posters. find
  10. Kauchak, D., & Barzilay, R. (2006). Paraphrasing for automatic evaluation. Paper presented at the Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. our card
  11. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. Paper presented at the MT summit. find
  12. Koehn, P. (2010). Statistical Machine Translation: Cambridge University Press. ↗
  13. Kumano, T., Tanaka, H., & Tokunaga, T. (2007). Extracting phrasal alignments from comparable corpora by using joint probability smt model. Proceedings of TMI. find
  14. Marton, Y., Callison-Burch, C., & Resnik, P. (2009). Improved statistical machine translation using monolingually-derived paraphrases. Paper presented at the Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. our card
  15. Munteanu, D. S., & Marcu, D. (2002). Processing comparable corpora with Bilingual Suffix Trees. Paper presented at the Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10. find
  16. Munteanu, D. S., & Marcu, D. (2005). Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. Comput. Linguist., 31(4), 477-504. doi: 10.1162/089120105775299168 ↗ our card
  17. Munteanu, D. S., & Marcu, D. (2006). Extracting parallel sub-sentential fragments from non-parallel corpora. Paper presented at the Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. our card
  18. Nakov, P. (2008). Paraphrasing verbs for noun compound interpretation. Paper presented at the Proc. of the LREC Workshop Towards a Shared Task for MWEs (MWE 2008). find
  19. Och, F. J., & Ney, H. (2002). Discriminative training and maximum entropy models for statistical machine translation. Paper presented at the Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. our card
  20. Rapp, R. (1999). Automatic identification of word translations from unrelated English and German corpora. Paper presented at the Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics, College Park, Maryland. our card
  21. Resnik, P., & Smith, N. A. (2003). The Web as a parallel corpus. Comput. Linguist., 29(3), 349-380. doi: 10.1162/089120103322711578 ↗ our card
  22. Saad, M., Langlois, D., & Smaïli, K. (2013). Extracting comparable articles from wikipedia and measuring their comparabilities. Procedia-Social and Behavioral Sciences, 95, 40-47. find
  23. Sharoff, S., Babych, B., & Hartley, A. (2006). Using comparable corpora to solve problems difficult for human translators. Paper presented at the Proceedings of the COLING/ACL on Main conference poster sessions. find
  24. Skadiņa, I., Aker, A., Mastropavlos, N., Su, F., Tufis, D., Verlic, M., . Gaizauskas, R. (2012). Collecting and using comparable corpora for statistical machine translation. Paper presented at the Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC), Istanbul, Turkey. find
  25. Skadiņa, I., Vasiļjevs, A., Skadiņš, R., Gaizauskas, R., Tufiş, D., & Gornostay, T. (2012). Analysis and evaluation of comparable corpora for under resourced areas of machine translation. Paper presented at the The 5th Workshop on Building and Using Comparable Corpora. find
  26. Zhao, S., Niu, C., Zhou, M., Liu, T., & Li, S. (2008). Combining Multiple Resources to Improve SMT-based Paraphrasing Model. Paper presented at the ACL. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abouammoh-2016-creation-comparable-corpora-english-urdu/