Elamur Research⌕
☰
home / papers / 2023 / How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction

How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction

Paper · 2023 · Mohan Raj Chanthran, Lay-Ki Soon, Huey Fang Ong et al.

Summary

The paper asks whether ChatGPT can extract named entities and relations from Malaysian English, a variety shaped by Bahasa Malaysia, Chinese and Tamil. It borrows local words, coins blends and forms derivations such as honorifics like "Tan Sri", Malay place names inside compound names, and demonyms like "Selangorians". The authors built the MEN-Dataset: 200 news articles from New Straits Times, Malay Mail and Bernama, annotated with OntoNotes-style entity labels and relation labels taken from ACE05 and DocRED. They then tested ChatGPT through its web interface with a pipeline they call educate-predict-evaluate. ChatGPT is given the annotation guideline, few-shot examples with annotator explanations and self-consistency voting, and its output is scored against human labels with F1 across 18 prompt settings. Entity extraction was weak, peaking at an F1 of 0.497. A comparison with DocRED, which is Standard English, showed NER consistently worse on the Malaysian data, while relation extraction scores were similar on both. The authors attribute the NER gap to local word-formation patterns. The work sets a first benchmark for LLM information extraction on this English variety.

Key points

  • MEN-Dataset has 200 articles, 11 entity labels and 101 relation labels. It contains 6,061 entity mentions (2,874 unique) and 3,268 relations: 2,237 under DocRED-style labels and 1,031 under ACE-2005-style labels.
  • 18 prompt settings were compared. They vary zero-shot vs. few-shot, with or without the guideline (ICL), with or without explanations, and the number of repeated iterations. The best entity F1 on MEN was only 0.497.
  • Self-consistent prompting sends each article to ChatGPT several times (e.g., three) and keeps only entities that recur across runs.
  • The cross-dataset test used only the PERSON, ORGANIZATION and LOCATION labels and the 84 relation types the two datasets share.
    • NER F1 was about 0.56–0.57 on MEN versus 0.615–0.738 on DocRED.
    • RE F1 was about 0.58–0.70 on MEN versus 0.64–0.76 on DocRED, with no consistent winner.
  • Experiments ran in the ChatGPT web UI rather than the API, because the web chat keeps the guideline in conversation memory. The code is released on GitHub (mohanraj-nlp/ChatGPT-Malaysian-English).

Interesting ideas

  • I1. Giving the LLM the human annotation guideline itself, not just labeled examples, helps align its output with the dataset's labeling conventions.
  • I2. Few-shot examples come with short annotator explanations of why each span was tagged, so the model gets the reason behind each label as well as the label.
  • I3. Running the same prompt several times and keeping only the answers that recur is a cheap way to filter noise from a stochastic extractor.
  • I4. The evaluation compares a dialect dataset with a standard-English one on only the overlapping labels and relations, so the gap reflects the language variety rather than mismatched schemas.
  • I5. The error analysis sorts failures by word-formation type: loan words, compound blends and derived words. These map to concrete cases such as honorific titles inside names, Malay place words inside venue names, and demonyms built from state names.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could be prompted with Mova's own UD and frame annotation conventions, so that silver labels follow our scheme rather than the teacher's habits.
  • I2 (learn): teacher prompts and stored silver data could carry a one-line justification per label, which also feeds Mova's explainability and makes self-induced repair rules easier to audit.
  • I3 (learn): repeated teacher samples can join the witness parsers as extra voters, with only agreed entities, arcs or frames admitted to silver data.
  • I4 (learn): when testing robustness on non-standard English or converted treebanks, first reduce both sides to a shared label set, then compare.
  • I5 (tokens): the tagger and lemmatizer could tag honorific spans as part of PERSON names and map demonyms like "Selangorians" to their place lemma, so later stages can link people to locations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash University MalaysiaSubang Jaya · Malaysia4 works · 10 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 worksInformation extraction · 167 worksNamed entity recognition · 200 works

Cite

Cite as Mohan Raj Chanthran, Lay-Ki Soon, Huey Fang Ong and Bhawani Selvaretnam. 2023. How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction. Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chanthran2023how,
  title = {{How well ChatGPT understand Malaysian English? An Evaluation on Named Entity Recognition and Relation Extraction}},
  author = {Mohan Raj Chanthran, Lay-Ki Soon, Huey Fang Ong and Bhawani Selvaretnam},
  booktitle = {Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)},
  year = {2023},
  url = {https://aclanthology.org/2023.gem-1.30/}
}

References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Belal, James She, and Simon Wong. 2023. Leveraging chatgpt as text annotation tool for sentiment analysis. ArXiv, abs/2306.17177. ↗
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005.14165. ↗
  3. Chunkit Chan, Cheng Jiayang, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. ArXiv, abs/2304.14827. ↗
  4. Lingjiao Chen, Matei Zaharia, and James Y. Zou. 2023. How is chatgpt’s behavior changing over time? ArXiv, abs/2307.09009. ↗
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1–240:113. ↗
  6. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America, 120. ↗
  7. Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. ArXiv, abs/2305.14450. ↗
  8. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, New York City, USA. Association for Computational Linguistics. ↗
  9. Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li, Xiaoqian Jiang, and Hua Xu. 2023. Zero-shot clinical entity recognition using chatgpt. ArXiv, abs/2303.16416. ↗
  10. T.S. Imm. 2014. Exploring the malaysian english newspaper corpus for lexicographic evidence. 32:167–185. find
  11. Noriah Ismail, Normah Ismail, and Kamalanathan Ramakrishnan. 2007. Malaysian english versus standard english: Which is favored? find
  12. Xiaoxi Kang, Lizhen Qu, Lay-Ki Soon, Adnan Trakic, Terry Yue Zhuo, Patrick Charles Emerton, and Genevieve Grant. 2023. Can chatgpt perform reasoning using the irac method in analyzing legal scenarios like a lawyer? arXiv preprint arXiv:2310.14880. ↗
  13. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. ArXiv, abs/2304.11633. ↗
  14. OpenAI. 2022. chatgpt. Chatgpt. find
  15. OpenAI. 2023. 2303.08774. Gpt-4. https://openai.com/https://arxiv.org/abs/ ↗
  16. Michael V. Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. ArXiv, abs/2304.11085. ↗
  17. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288. ↗
  18. Christopher Walker. 2005. Multilingual Training Corpus LDC2006T06. Web Download. Philadelphia: Linguistic Data Consortium. find
  19. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171. ↗
  20. Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia. 2023. Is chatgpt a good sentiment analyzer? a preliminary study. ArXiv, abs/2304.04339. ↗
  21. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903. ↗
  22. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-shot information extraction via chatting with chatgpt. ArXiv, abs/2302.10205. ↗
  23. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 764–777, Florence, Italy. Association for Computational Linguistics. *Figure 4: Prompt template used to provide entity annotation guideline as separate chunks* *Figure 5: A few examples of manually annotated entities along with explanations for why they have been annotated.* *Figure 6: The prompt template used to extract entities based on news article provided.* *Figure 7: The prompt template used to extract relations based on news article and entities provided.* ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chanthran-2023-how-well-chatgpt-understand-malaysian/