Elamur Research⌕
☰
home / papers / 2024 / Limpeh ga li gong: Challenges in Singlish Annotations

Limpeh ga li gong: Challenges in Singlish Annotations

Paper · 2024 · Luo Qi Chan and Lynnette Hui Xian Ng

Summary

The paper examines how well standard English NLP tools handle Singlish (Colloquial Singapore English). Singlish draws vocabulary and grammar from Malay, Hokkien, Mandarin and other languages, and it has no fixed spelling. The authors focus on part-of-speech tagging as a basic test. They took 92 SMS messages from the NUS-SMS-Corpus. Two native speakers translated them into standard English with minimal edits and checked each other's work, and one native speaker hand-tagged the Singlish originals with POS labels. Two off-the-shelf spaCy taggers were scored against these gold labels. Both reached only about 80% accuracy, far below their reported figures on regular English. A qualitative analysis groups the failures into four kinds:

  • unstable forms and meanings,
  • discourse particles whose meaning depends on context,
  • constructions specific to Singlish,
  • variation across speaker groups.

The work matters because it documents a clear domain gap for a colloquial English variety. It also publishes a small parallel, annotated resource and shows why annotating such varieties requires deliberate, partly arbitrary linguistic decisions.

Key points

  • Data: 92 sentences sampled after filtering the roughly 55k-message NUS-SMS-Corpus by a length threshold of 100. The release contains three layers: raw Singlish, a parallel English translation, and manual plus automatic POS tags (github.com/luoqichan/singlish).
  • Taggers: spaCy en_core_web_sm (transition-based) scored 0.80 accuracy and en_core_web_trf (fine-tuned roberta-base) scored 0.82. The documented English scores for the same models are 0.97 and 0.99.
  • Dominant error: words whose lemma looks non-standard get pushed toward NOUN or PROPN. The particle "ba" became NOUN, "leh" got NOUN or INTJ depending on the model, and "le" (from Chinese 了) got INTJ or X.
  • Lexical traps for the taggers:
    • "go down" means simply "go", calqued from Chinese 下, so "down" is not an adposition.
    • "like" meaning "approximately" was hand-tagged ADJ.
    • Spellings vary freely, e.g. lah/la, lor/lorh, and "oredi" for "already".
    • Copulas and auxiliaries are often dropped, as in "The bus coming".
  • In a manual UD tree for the denial construction "where got", the annotators attached it as an aux to the clause head, by analogy with "can". They openly call this choice arbitrary.

Interesting ideas

  • I1. Pairing each raw colloquial sentence with a minimally edited standard-English rendering gives an alignment that separates dialect-specific structure from shared content, which makes contrastive analysis easy.
  • I2. The paper finds that taggers quietly fall back to NOUN or PROPN on unfamiliar lemmas, so the error has a recognizable signature that can be detected.
  • I3. Sentence-final particles are emotive and carry no syntactic link to the clause, so the authors label them interjections rather than forcing them into the clause structure.
  • I4. To justify a tree for an odd construction, the authors line up several meaning-equivalent paraphrases and choose the head and relation that best match the shared reading.
  • I5. The domain gap is measured by scoring the same models on out-of-domain gold data and comparing the result with their published in-domain numbers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Raw–standard pairs like these could serve as test material for UD dialect conversion. The parse of the normalized side acts as a reference tree for checking the converted parse of the raw side.
  • I2 (tokens): The tagger could mark out-of-lexicon tokens tagged NOUN/PROPN as low-confidence and expose that flag in its explanation, so later stages do not build frames on a guessed noun.
  • I3 (frames): Particles like "lah", "meh" or "one" could be routed to speech-act and stance features, such as certainty, slight or invitation, rather than to semantic roles.
  • I4 (tree): When a new construction has no clear UD analysis, a paraphrase set could decide its attachment and be stored as the documented rationale for that conversion rule.
  • I5 (learn): A small native-annotated set in a colloquial variety could become a fixed out-of-domain control in evaluation, catching gains that hold only on standard treebank English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Luo Qi Chan and Lynnette Hui Xian Ng. 2024. Limpeh ga li gong: Challenges in Singlish Annotations. arXiv:2410.16156.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.16156 · arXiv 2410.16156

@misc{chan2024limpeh,
  title = {{Limpeh ga li gong: Challenges in Singlish Annotations}},
  author = {Luo Qi Chan and Lynnette Hui Xian Ng},
  year = {2024},
  doi = {10.48550/arxiv.2410.16156},
  eprint = {2410.16156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.16156}
}

References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Steven Adam. 2022. "in the past, we hear that a lot": Features of and responses to tense and aspect in written singaporean academic english. English for Specific Purposes, 65:107–119. find
  2. Michele Banko and Robert C Moore. 2004. Part-of-speech tagging in context. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 556–561. find
  3. Eric Brill. 1992. A simple rule-based part of speech tagger. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992.
  4. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740–750, Doha, Qatar. Association for Computational Linguistics. ↗
  5. Tao Chen and Min-Yen Kan. 2012. Creating a live, public short message service corpus: The nus sms corpus. Language Resources and Evaluation. ↗
  6. Alebachew Chiche and Betselot Yitagesu. 2022. Part of speech tagging: a systematic review of deep learning and machine learning approaches. Journal of Big Data, 9(1):10. ↗
  7. David Deterding. 2007. Singapore English. Edinburgh University Press. find
  8. Linus Tze En Foo and Lynnette Hui Xian Ng. 2024. Disentangling singlish discourse particles with task-driven representation. arXiv preprint arXiv:2409.20366. ↗
  9. Héctor Goldar Perrote et al. 2014. Sentence structure of singlish in written materials: The influence of chinese on the english superstrate. find
  10. Wilkinson Daniel Wong Gonzales, Mie Hiramoto, Jakob RE Leimgruber, and Jun Jie Lim. 2023. The corpus of singapore english messages (cosem). World Englishes, 42(2):371–388. find
  11. Anthea Fraser Gupta. 2006. Epistemic modalities and the discourse particles of singapore. In Approaches to discourse particles, pages 243–263. Brill. ↗
  12. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. find
  13. Julian Kupiec. 1992. Robust part-of-speech tagging using a hidden markov model. Computer speech & language, 6(3):225–242. ↗
  14. Koh En Hui Lauren. 2018. An analysis of colloquial singapore english “where got” constructions. find
  15. Jakob RE Leimgruber. 2013. Singapore English: Structure, variation, and usage. Cambridge University Press. find
  16. Li Lin, Kunmei Han, Jia Wen Hing, Luwen Cao, Vincent Ooi, Nick Huang, and Zhiming Bao. 2023. Tagging singapore english. World Englishes, 42(4):624– 641. ↗
  17. Zhengyuan Liu, Shikang Ni, Aiti Aw, and Nancy Chen. 2022. Singlish message paraphrasing: A joint task of creole translation and text normalization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3924–3936. find
  18. Chengyao Lv, Huihua Liu, Yuanxing Dong, and Yunliang Chen. 2016. Corpus based part-of-speech tagging. International journal of speech technology, 19:647–654. ↗
  19. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. ↗
  20. Alastair Pennycook. 2006. Global Englishes and transcultural flows. routledge. find
  21. ICE Singapore. ICE-Singapore — ice-corpora.uzh.ch. https://www.ice-corpora.uzh.ch/en/[Accessed 17-joinice/Teams/icesin.html. 09-2024]. ↗
  22. Samira Tasharofi, Fahimeh Raja, Farhad Oroumchian, and Masoud Rahgozar. 2007. Evaluation of statistical part of speech tagging of persian text. In 2007 9th International Symposium on Signal Processing and Its Applications, pages 1–4. IEEE. find
  23. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 human language technology conference of the north american chapter of the association for computational linguistics, pages 252–259.
  24. UniversalDependencies. Universal Dependencies — universaldependencies.org. https://universaldependencies.org/#language-u. [Accessed 21-10-2024]. ↗
  25. Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014. Pos tagging of english-hindi code-mixed social media content. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 974–979.
  26. Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Aiti Aw, and Nancy Chen. 2024. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 370–390. ↗
  27. Jock Wong. 2005. "why you so singlish one?" a semantic and cultural interpretation of the singapore english particle one. Language in Society, 34(2):239–275. find
  28. Jock O Wong. 2014. The culture of Singapore English. Cambridge University Press. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chan-2024-limpeh-ga-li-gong-challenges/