Elamur Research⌕
☰
home / papers / 2022 / A comparison of the validity of measurement methods of the general English proficiency by dictation and read-aloud performance

A comparison of the validity of measurement methods of the general English proficiency by dictation and read-aloud performance

Paper · 2022 · Katsunori Kotani and Takehiko Yoshimi

Summary

Commercial English tests such as TOEIC or TOEFL are costly, take hours, and use material unrelated to what a class is studying. That makes them a poor fit for frequent in-class checks of general English proficiency (GEP). Kotani and Yoshimi ask whether cheap classroom tasks can stand in for such tests. They compare dictation, reading aloud, or both together as the basis for estimating GEP.

Fifty Japanese university students each worked through 15 sentences from two Aesop fables, giving 750 sentence-level instances. For each sentence the authors recorded:

  • dictation scores: accuracy and self-rated ease;
  • read-aloud scores: accuracy, speech rate and self-rated ease;
  • five linguistic difficulty features of the sentence.

Regression models predicted each learner's TOEIC Listening & Reading score and were checked by cross-validation. Combining dictation and read-aloud gave the closest match to observed TOEIC scores (r = 0.80). Dictation alone reached 0.75 and read-aloud alone 0.59, and all pairwise gaps were statistically significant. The practical message is that teachers can estimate proficiency from simple audio or text materials of their own choosing. Dictation alone is a reasonable lighter option.

Key points

  • The data are 50 learners × 15 sentences = 750 instances. The texts are "The North Wind and the Sun" (5 sentences) and "The Boy Who Cried Wolf" (10 sentences), read by a native American English voice actor.
  • The performance features are EASE-D and ACC-D for dictation, and EASE-R, ACC-R and RATE-R for read-aloud. The sentence features are length, mean word length, number of multi-syllable words, word difficulty and speech rate.
  • The target is the TOEIC Listening & Reading score: mean 607.7, range 295–900, SD 184.45, approximately normally distributed.
  • Correlations with observed GEP were D&R 0.80, D 0.75 and R 0.59, all significant.
  • Chi-square tests for equal correlations, with FDR correction, found all three gaps significant: D&R > D (4.89), D&R > R (65.79) and D > R (34.78).

Interesting ideas

  • I1. Two weak signals that probe different skills (spelling through listening, pronunciation through reading) combine into a stronger estimate than either alone, and the authors confirm the gain with a formal test of correlation equality rather than eyeballing the numbers.
  • I2. Learners' own rating of how easy each item felt is used as a predictor next to objective accuracy, treating self-assessment as cheap extra evidence.
  • I3. Each item carries simple linguistic difficulty features (sentence length, multi-syllable word count, word difficulty), so the same raw score can be read relative to how hard the sentence was.
  • I4. Test texts were picked to cover the basic sound inventory of English, with the second fable filling sounds the first one lacks.
  • I5. The authors weigh accuracy against preparation and administration cost and endorse a cheaper single-task method when its loss is modest.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when comparing parser or reader variants, test pairwise differences in agreement scores for significance with FDR correction, so small gains from combining witnesses are not overclaimed.
  • I2 (answer): student readers could report a per-question confidence alongside their answer, and pairing it with correctness would give a better estimate of how well a course is really mastered.
  • I3 (answer): tag school-test and fable QA items with difficulty features computed from the en crate, then report accuracy adjusted for item difficulty instead of a flat average.
  • I4 (learn): build evaluation sets by covering an inventory, for example all UD relations or VerbNet classes, so that each added text closes specific gaps left by the others.
  • I5 (learn): report every evaluation configuration together with its cost, such as LLM teacher calls or GPU time, so cheaper tiers can be chosen deliberately when their accuracy gap is small.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Katsunori Kotani and Takehiko Yoshimi. 2022. A comparison of the validity of measurement methods of the general English proficiency by dictation and read-aloud performance. Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{kotani2022comparison,
  title = {{A comparison of the validity of measurement methods of the general English proficiency by dictation and read-aloud performance}},
  author = {Katsunori Kotani and Takehiko Yoshimi},
  booktitle = {Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation},
  year = {2022},
  url = {https://aclanthology.org/2022.paclic-1.43/}
}

References

27 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoav Benjamini and Yosef Hochberg.1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B (Methodological), 57(1):289‒300. find
  2. Jeanne S. Chall and Harold E. Dial. 1948. Predicting Listener Understanding and Interest in Newscasts. Educational Research Bulletin, 27(6):141‒153+168. find
  3. David Coniam. 1991. Reading Aloud Speed as a Factor in Oral Fluency and General Language Proficiency? Hongkong Papers in Linguistics and Language Teaching, 14:47‒69. find
  4. David Deterding. 2006. The North Wind versus a Wolf: Short Texts for the Description and Measurement of English Pronunciation. Journal of the International 36(2):187‒196. find
  5. Phonetic Association, Educational Testing Service. 1998. TOEIC Technical Manuel. Educational Testing Service, Princeton: NJ. find
  6. Irving E. Fang. 1966. The Easy Listening Formula. Journal of Broadcasting, 11(1):63‒68. find
  7. Jiang Fu, Yuya Chiba, Takashi Nose, and Akinori Ito. 2020. Automatic Assessment of English Proficiency for Japanese Learners without Reference Sentences based on Deep Neural Network Acoustic Models. Speech Communication, 116:86‒97. find
  8. Atsushi Iino, Yukiko Yabuta, and Joel Thomas. 2011. Relationship between Criteria for Reading Aloud Evaluation and English Proficiency. Journal of the Chubu English Language Education, 40:159‒166. find
  9. International Phonetic Association. 1999. Handbook of the International Phonetic Association: A Guide to the Use of the International Phonetic Alphabet. Cambridge University Press, Cambridge, UK. find
  10. Patricia Irvine, Parvin Atai, and John W. Oller, Jr. 1974. Cloze, Dictation, and the Test of English as a Foreign Language. Language Learning, 24(2):245‒252. find
  11. Masaya Kanzaki. 2015. Minimal English Test: Item Analysis and Comparison with TOEIC Scores. Shiken: JALT Testing & Evaluation SIG newsletter, 19(2):12‒23. find
  12. Semin Kazazoglu. 2013. Dictation as a Language Learning Tool. Procedia-Social and Behavioral Sciences, 70:1338‒1346. find
  13. Hideo Kiyokawa. 1990. A Formula for Predicting Listenability: The Listenability of English Language Materials 2. Wayo Women's University Language and Literature, 24:57‒74. find
  14. Katsunori Kotani and Takehiko Yoshimi. 2021a. Predicting English Proficiency with Read-Aloud Performance and Linguistic Difficulty of Sentences. Proceedings of International Technology, Education and Development Conference, pages 1180‒1185. find
  15. Katsunori Kotani and Takehiko Yoshimi. 2021b. Prediction of General ESL Proficiency Considering Learners' Dictation Performance. The 3rd ETLTC International Conference on Information and Communications Technology, EDP Sciences, France. find
  16. Satsuki Kojima and Soichi Ota. 2012. Shadowing, Dictation and Reading Aloud: Which is Effective? Journal of The Japan Association of College English Teachers, 4:29‒40. find
  17. Yongeun Lee. 2014. Quantifying English Fluency in Korean Speakers' Read-Aloud and Picture-Cued Storytelling Speech. Linguistic Research, 31(3):465‒490. find
  18. Paul Leeming and Aeric Wong. 2016. Using Dictation to Measure Language Proficiency: A Rasch Analysis. Language Testing and Assessment, 5(2):1‒25. find
  19. David Meyer, Evgenia Dimitriadou, Kurt Hornik, Andreas Weingessel, Friedrich Leisch, Chih-Chung Chang, Chih-Chen Lin. 2021. e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. find
  20. Jose Noijons. 1994. Testing Computer Assisted Language Testing: Towards a Checklist for CALT. CALICO Journal, 12(1):37‒58. find
  21. John W. Oller, Jr. 1983. Evidence for a General Language Proficiency Factor: An Expectancy Grammar. In John W. Oller, Jr. (Ed.), Issues in Language Testing Research, pages 3‒10, Newbury House, Rowley: MA. find
  22. Henry L. Roediger III, Adam L. Putnam, and Megan A. Smith. 2011. Ten Benefits of Testing and their Applications to Educational Practice. In Jose P. Mestre and Brian H. Ross (Eds.), Psychology of Learning and Motivation: Cognition in Education, 55: 1‒36, San Diego, CA: Elsevier Academic Press. find
  23. Anthony Stenton. 2013. The Role of the Syllable in Foreign Language Learning: Improving Oral Production through Dual-Coded, Sound-Synchronised, Typographic Annotations. Language Learning in Higher Education. Journal of the European Confederation of Language Centres in Higher Education, 2(1):145‒161. find
  24. Ruslan Suvorov and Volker Hegelheimer. 2013. Computer-Assisted Language Testing. In Antony J, Kunnan (Ed.), The Companion to Language Assessment, John Wiley & Sons, Hoboken: NJ. find
  25. Elvis Wagner. 2020. Duolingo English Test, Revised Version July 2019, Language Assessment Quarterly, 17(3):300‒315. find
  26. Aeric Wong and Paul Leeming. 2014. Using Dictation to Measure Language Proficiency. Language Education in Asia, 5(1):160‒169. find
  27. Anoushe Yazdinejad and Mitra Zeraatpishe. 2019. Investigating the Validity of Partial Dictation as a Test of Overall Language Proficiency. International Journal of Language Testing, 9(2):44‒55. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kotani-2022-comparison-validity-measurement-methods-general/