Elamur Research⌕
☰
home / papers / 2002 / EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation

EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation

Summary

The paper reports on the interim state of the EMILLE (Enabling Minority Language Engineering) project of the Universities of Lancaster and Sheffield. The project aims to build a large corpus of South Asian languages and an environment for processing them automatically. The authors first planned written, spoken and parallel corpora for seven languages. Electronic texts in Indic scripts were scarce, and OCR for these scripts was still immature. The data were therefore collected by hand from news websites that used a single downloadable 8-bit ttf font, with permission obtained from the publishers. These fonts use glyph encodings that are incompatible with one another, so a separate conversion table to Unicode had to be written for each source. Markup followed the CES standard. A collaboration with the Indian CIIL extended coverage to 14 languages and added more genres. In parallel, the authors developed the GATE architecture, above all its Unicode support, and designed an EAGLES-based part-of-speech tagset for Urdu. For low-resource languages, this work is a useful example of how to build a corpus when texts are spread across incompatible encodings.

Key points

  • Initial plan: at least 9 million words of written text for each of Bangla, Gujarati, Hindi, Panjabi, Singhalese, Tamil and Urdu. For the five languages with large communities in the UK, they also planned 500 thousand words of spoken language and 200 thousand words of parallel translations from English for each.
  • As of April 2002, the EMILLE/CIIL corpus contained about 60.9 million words in 14 languages. The largest parts were Tamil (10.1 million), Hindi (8.8 million) and Gujarati (7.8 million).
  • The authors found no Unicode texts in Indian languages on the web. Websites published texts as images, as PDF, through dedicated programs (e.g., Urdu 98), in 8-bit ttf fonts, or in embedded fonts. Only the ttf-font option proved suitable for collection. Font developers effectively ignored the ISCII standard.
  • The Urdu tagset was built within the EAGLES framework. Gender, number and case fit this scheme almost unchanged (the oblique case was labelled as dative). Difficulties arose with verbal categories and with language-specific phenomena, such as the "zimmah" problem in borrowed vocabulary.
  • The authors later plan to release a POS tagger for Urdu, font conversion programs, and Unicode sentence alignment tools.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{baker2002emille,
  title = {{EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation}},
  author = {Baker, Paul and Hardie, Andrew and McEnery, Tony and Cunningham, Hamish and Gaizauskas, Rob},
  booktitle = {Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02)},
  year = {2002},
  doi = {10.63317/24zr5nmbm3t3},
  url = {https://doi.org/10.63317/24zr5nmbm3t3}
}

References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baker, J. P., Burnard, L., McEnery, A. M. & Wilson, A. (1998), Techniques for the Evaluation of Language Corpora: a report from the front. Proceedings of the First International Conference on Language Resources and Evaluation: Granada. find
  2. Baker, J.P., Lie. M., McEnery, A.M. and Sebba, M. 2000. Building a Corpus of Spoken Sylheti. Literary and Linguistic Computing, 15:419-431. find
  3. Bureau of Indian Standards, Indian Standard Code for Information Interchange, IS13194, 1991. find
  4. Crowdy, S. 1995. The BNC spoken corpus. In G. Leech, G. Myers and J. Thomas (eds.), Spoken English on computer: transcription, mark-up and application. Longman:London. find
  5. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V. and Wilks, Y. 2000. Experience of using GATE for NLP R&D. In Proceedings of the Workshop on Using Toolsets and Architectures To Build NLP Systems at COLING-2000:Luxembourg. find
  6. Leech, G. and Wilson, A. (1999) Standards for tagsets. In van Halteren, H. (ed.), Syntactic Wordclass Tagging. Kluwer:Dordrecht. find
  7. McEnery, A., Baker, J.P. and Burnard, L. 2000. ‘Corpus Resources and Minority Language Engineering’, in M. Gavrilidou, G. Carayannis, S. Markantontou, S. Piperidis and G. Stainhauoer (eds) Proceedings of the Second International Conference on Language Resources and Evaluation:Athens. find
  8. McEnery, A.M. & Ostler, N. 2000. ‘A New Agenda for Corpus Linguistics – Working With All of the World’s Languages’, Literary and Linguistic Computing, 15:401-418. find
  9. Oksefjell, S. 1999. A Description of the English-Norwegian Parallel Corpus: Compilation and Further Developments. International Journal of Corpus Linguistics, 4:197-219. find
  10. Pal, U. and Chaudhuri, B.B. 1995. Computer recognition of printed Bangla script. International Journal of System Science, 26:2107-2123. find
  11. Schmidt, R. L. 1999. Urdu: an essential grammar. Routledge:London. find
  12. Singh, S., McEnery, A. and Baker, J.P. 2000. Building a Parallel Corpus of English/Panjabi. In J. Veronis (ed.), Parallel Text Processing, Kluwer:Dordrecht. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/baker-2002-emille-67-million-word-corpus/