📄 Performance Comparison of Large Language Models on VNHSGE English Dataset: OpenAI ChatGPT, Microsoft Bing Chat, and Google Bard

Summary

Since ChatGPT was not officially offered in Vietnam while Bing Chat and Google Bard were, the author asks whether these alternatives are equally useful for high-school English study. Three chatbots were tested zero-shot on 250 multiple-choice items from the VNHSGE English dataset. The items come from five national graduation exams (2019–2023) and cover pronunciation and stress, grammar and vocabulary, communication, cloze reading and reading comprehension. Each question was wrapped in a fixed template asking for a letter choice plus an explanation, and answers were graded as simply right or wrong. BingChat scored highest, followed by Bard and then ChatGPT (GPT-3.5). An oracle that takes the best answer among the three per question did better still. All models failed the opening phonetics items. Converted to the 10-point national scale, every model beat the average Vietnamese candidate. The author concludes that BingChat and Bard are workable substitutes for ChatGPT and promising aids for English learning. This matters as a localized, exam-grounded check of chatbots in a specific education system.

Key points

• Data: the VNHSGE English dataset has 2500 multiple-choice questions. The evaluation used 250 of them, which are five exams of 50 items each, one per year from 2019 to 2023.
• Accuracy: BingChat 92.4%, Bard 86%, ChatGPT 79.2%. The best-of-three oracle (LLM_B) reached 97.2%.
• On the 10-point scale, the average scores were ChatGPT 7.92, BingChat 9.24 and Bard 8.6. For comparison, students in 2023 had an average of 5.84 and a most frequent score of 3.8.
• All three models answered the first four questions (pronunciation and stress) incorrectly. They did well on vocabulary, grammar and reading.
• Year-to-year results were fairly flat for ChatGPT, Bard and LLM_B. BingChat and the worst-of-three case (LLM_W) varied more between years.

Interesting ideas

• I1. The paper scores the per-item maximum and minimum across several systems (LLM_B and LLM_W), which gives an upper and lower bound on what the ensemble could achieve; this shows how much complementarity exists before any voting scheme is built.
• I2. Results are broken down by question position, which in these exams equals question type, so a whole skill blind spot (phonetics) becomes visible that a single overall average would hide.
• I3. The prompt requires a fixed answer format with a discrete choice plus a justification, which makes grading automatic while keeping a rationale for inspection.
• I4. Machine scores are placed on the official human score distribution (both the mean and the mode), instead of being reported as a bare percentage, which makes them meaningful to educators.
• I5. Stability is checked across exam years as a rough robustness test against drift in item style.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Computing best-of and worst-of agreement across witness parsers would show how much headroom silver-data voting actually has, and where every witness fails together.
• I2 (answer): School-test readers should report accuracy per section or item type, so that gaps such as stress and pronunciation items surface as separate failure classes.
• I3 (learn): LLM teacher tiers could be held to a strict "label + explanation" schema, so labels are machine-checkable and explanations can be audited against Mova's own derivation.
• I4 (answer): Results on school-course tests could be reported against published student score distributions, to show where Mova stands relative to real learners.
• I5 (learn): Evaluation could be sliced by source year or edition of test material, to detect rules or models that overfit one exam generation.