📄 Same Spelling, Different Functions Mah:Evaluating Language Models' Understanding of Singlish Particles

Summary

Singlish sentence-final particles such as lah, hor or leh change their pragmatic function depending on tone. Writing does not mark tone, so a text-only model has to work out the intended function from context. This paper asks how well language models manage that.

The author mined the English part of the NUS-SMS corpus with regular expressions and kept 10 frequent, well-documented particles. Instances of tonally ambiguous particles were hand-labelled with their most likely functional variant. The particle functions were then split into four semantic groups, chosen so that no group holds near-synonymous options. The test is a cloze task: the particle is masked and the model must pick the right one from its group.

Models tested:

• BERT-base-uncased
• BERT-base-multilingual-uncased
• SingBERT
• GPT-4.1 with zero-shot, definition-based and few-shot prompts

Domain pretraining helped clearly: SingBERT reached 56.2% against 30.1% for general English BERT. Still, no setup did well in absolute terms. This suggests context cannot fully stand in for prosody, which matters for any contact language.

Key points

• The data come from the NUS-SMS corpus (over 55K messages). After cleaning (deduplication, dropping anonymization tokens, removing messages under three words, excluding pure substrate-language text), 4,910 particle occurrences remained, across 10 particles and 14 tone/function variants.
• Four particles show tonal polysemy. one and what were left out because they share spelling with ordinary English words.
• SingBERT scored 56.2% overall versus 30.1% for BERT-base. GPT-4.1 ranged from 23.8% to 66.4% depending on group and prompt type.
• For GPT-4.1, definition prompting averaged 53.7%, few-shot 49.5% and zero-shot 45.0%. In Group 3, definitions raised accuracy from 23.8% to 46.7%, while few-shot reached only 30.0%.
• Results by group:
  • Group 1 (question markers) was the easiest and most stable.
  • Group 2 defeated the general English models (15.0% and 18.9%).
  • Group 3 (mixed functions) kept every model below 47%.
  • Group 4 (persuasive/confirmatory) reached 60.4% with SingBERT.

Interesting ideas

• I1. Restricting the cloze choices to groups whose members do not overlap in function avoids punishing a model for confusing labels that are near-synonyms anyway, so evaluation stays fair when the label set is fuzzy.
• I2. One written form is split into several tone-specific function entries, and each corpus instance is assigned to the nearest variant using context and syntax (statement, imperative, question).
• I3. Hiding a discourse particle and asking the model to restore it is a cheap probe of whether pragmatic knowledge is present, without needing full semantic annotation.
• I4. Giving an LLM the label definitions worked better than giving it worked examples, and was much steadier across groups.
• I5. Particles that share spelling with standard words (one, what) were excluded deliberately to prevent false matches when extracting by surface form.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when testing speech-act or implicature labels, we could score inside such non-overlapping candidate groups so that real errors are separated from arguable confusions.
• I2 (frames): prag could store sentence-final tags like "right?" or "eh" as several speech-act variants and choose one from clause type, keeping the choice explainable.
• I3 (learn): masking tag questions or hedges in our texts and checking whether prag predicts the right stance-marker class would give a quick regression test for pragmatics.
• I4 (learn): prompts for the LLM teacher tiers should lead with explicit definitions of frames or speech acts rather than relying mostly on few-shot examples.
• I5 (tokens): a dialect-particle lexicon in the en crate should flag such homographs and leave them to the tagger and parser rather than matching them by string.