📄 Automating Dictionary Production: a Tagalog-English-Korean Dictionary from Scratch

Summary

The authors show how to create a Tagalog dictionary with English and Korean translations without having any ready-made lexicographic resources. Most of the work is done automatically. First, they compiled a 230-million-token Tagalog corpus from the web. The corpus was part-of-speech tagged and lemmatized, and a sketch grammar was written for Sketch Engine. Next, each part of the dictionary entry was generated automatically: the headword list based on frequency, word forms, sense division, collocations, synonyms, examples and illustrations. Senses were identified as follows: collocations from word sketches were grouped on the basis of FastText vectors. After that, editors in Lexonomy corrected the entries in stages, working on only one element of the entries at a time. Pronunciation was recorded manually. Of the 45,000 entries, the editors checked the 15,000 most frequent ones, and the rest remained automatic. The entire process took about nine months. The greatest difficulties came from managing the data and the editors' work, not from the quality of the automatic drafts. The work shows that the "automatic draft + manual post-editing" approach makes it possible to produce dictionaries for low-resource languages faster and more cheaply.

Key points

• Dictionary size. 45,000 entries: the editors manually checked the 15,000 most frequent ones, while 30,000 remained purely automatic. The dictionary was to be made freely available on the Naver platform (dict.naver.com).
• Corpus and annotation. The authors consider the 230-million-token Tagalog web corpus to be the largest for this language as of July 2019. Parts of speech were assigned by the Stanford PoS tagger, and lemmas by an improved Tagalog stemmer. The corpus is available in Sketch Engine.
• Sense identification. For each word, collocations with frequency ≥ 5 and logDice ≥ 2 were taken. For each collocation, the FastText vectors of the words within a window of up to 4 words were averaged, and the result was clustered with the HDBSCAN algorithm. Each sense rests on specific concordance lines, so examples and synonyms can be selected separately for each sense.
• Other entry components. Synonyms, antonyms and related words come from the Sketch Engine thesaurus: both in general and for a specific sense. Illustrations were sought in Wikimedia Commons, PixaBay and Google Image Search (only images with free licenses). Pronunciation was recorded manually, at roughly 900 words per eight-hour working day.
• Conclusions and plans. Automatic sense identification worked better than expected. At the same time, the authors stress that a larger corpus and a better tagger and lemmatizer would noticeably improve the results. Dictionaries for Urdu and Lao are already being prepared using the same approach.