Elamur Research⌕
☰
home / papers / 2019 / Rater attitude towards emerging varieties of English: a new rater effect?

Rater attitude towards emerging varieties of English: a new rater effect?

Paper · 2019 · Tammy Huei-Lien Hsu

Summary

The paper asks whether raters' attitudes towards a World Englishes (WE) variety affect the scores they give in a speaking test, which would be a new kind of rater effect. Hsu uses Indian English as the case. 96 teacher raters first scored six IELTS speech samples from Indian examinees on the four analytic IELTS criteria. They then filled in an online questionnaire, the "rater attitude instrument" (RAI), which measures how they view WE. The analysis used correlations between RAI scores and ratings, a MANOVA comparing attitude groups, Tukey contrasts, and a check of whether the RAI predicts scores. Raters with more favourable attitudes consistently gave higher scores than unfavourable ones on every criterion, and the correlations were moderate to strong. The RAI turned out to be a significant predictor of the scores awarded. This matters because IELTS formally accepts all standard varieties of English. A judge's hidden stance towards a variety can therefore threaten score fairness and validity. The author suggests treating this as fairness evidence to gather before testing, not only afterwards.

Key points

  • Participants and data: 96 teacher raters, six IELTS speech samples by Indian examinees, and the RAI (from Hsu 2016) completed online.
  • Statistics: correlation analysis, MANOVA across the four analytic rating criteria, and Tukey contrasts between attitude groups.
  • The group with the positive attitude scored higher than the group with the negative attitude on all four criteria.
  • Rater attitude accounted for roughly 30% of the variance in test scores.
  • Limitations named by the author:
    • no rigorous rater training;
    • Indian English was the only stimulus variety;
    • only descriptive tasks were used;
    • purely quantitative data;
    • a possible order effect, because raters' task order online was not enforced.

Interesting ideas

  • I1. Profile the judges separately from the items: give every evaluator an independent attitude instrument, then use that profile to predict and explain how they score.
  • I2. Compare evaluator groups criterion by criterion rather than on one overall score, since in earlier work (Kim 2005) the bias appeared in analytic scores but not in holistic ones.
  • I3. Keep the stimuli fixed and vary only the judge, so that any score difference can be attributed to the judge's stance and not to the input.
  • I4. Collect the judgments first and the attitude questionnaire afterwards to avoid priming, and treat an unenforced order as a confound.
  • I5. Treat fairness towards language varieties as evidence gathered before deployment, not as a post-hoc check.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Each witness parser or LLM teacher tier could get a measured "disposition" on non-standard English inputs, so systematic disagreement in silver data can be traced to a known judge bias instead of being averaged away.
  • I2 (learn): Evaluation reports could break results down by separate dimensions (attachment, labels, roles), so a bias that cancels out in a single headline metric still becomes visible.
  • I3 (learn): This works as a negative control, where the same sentences are labelled by different teachers and divergence beyond chance flags teacher-specific bias.
  • I4 (learn): When an LLM teacher both labels data and answers meta-questions about a dialect, the labelling should come first in a fixed, logged order.
  • I5 (tree): Before release, the parser could be tested on outer-circle English texts (for example Indian English) to see whether attachment accuracy drops for these varieties.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Fu Jen Catholic University.

📚 Shelves 2 Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Tammy Huei-Lien Hsu. 2019. Rater attitude towards emerging varieties of English: a new rater effect?. Language Testing in Asia 9.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1186/s40468-019-0080-0

@inproceedings{hsu2019rater,
  title = {{Rater attitude towards emerging varieties of English: a new rater effect?}},
  author = {Tammy Huei-Lien Hsu},
  booktitle = {Language Testing in Asia 9},
  year = {2019},
  doi = {10.1186/s40468-019-0080-0},
  url = {https://doi.org/10.1186/s40468-019-0080-0}
}

References

82 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeywickrama, P. (2013). Why not non-native varieties of English as listening comprehension test input? RELC Journal, 44, 59– 74. ↗
  2. Ajzen, I., & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs: Prentice Hall. find
  3. Ajzen, I., & Timko, C. (1986). Correspondence between health attitudes and behavior. Basic and Applied Social Psychology, 7(4), 259–276. ↗
  4. Albarracin, D., Johnson, B. T., & Zanna, M. P. (Eds.). (2005). Handbook of attitudes. Mahwah: Erlbaum. find
  5. Bayyurt, Y., & Altinmakas, D. (2012). A WE-based English communication skills course at a Turkish university. In A. Matsuda (Ed. ), Principles and practices of teaching English as an international language (pp. 169–182). Bristol: Multilingual Matters. ↗
  6. Berger, C. R., & Bradac, J. J. (1982). Language and social knowledge: Uncertainty in interpersonal relations. London: Arnold. find
  7. Bhatt, R. M. (2001). World Englishes [Electronic version]. Annual Review of Anthropology, 30, 527–550. find
  8. Brown, J. D. (2014). The future of World Englishes in language testing. Language Assessment Quarterly, 11, 5–26. ↗
  9. Browne, K., & Fulcher, G. (2017). Pronunciation and intelligibility in assessing spoken fluency. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 37–53). Bristol: Multilingual Matters. find
  10. Canagarajah, S. (2006). Changing communicative needs, revised assessment objectives: testing English as an international language. Language Assessment Quarterly, 3(3), 229–242. ↗
  11. Cargile, A. C., Giles, H., Ryan, E. B., & Y Bradac, J. J. (1994). Language attitudes as a social process: A conceptual model and new directions. Language & Communication, 14, 211–236. ↗
  12. Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 16–33. ↗
  13. Chalhoub-Deville, M., & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24(3), 383–391. ↗
  14. Cluver, A. D. (2000). Changing language attitudes: The stigmatization of Khoekhoegowap in Namibia. Language Problems and Language Planning, 24(10), 77–100. Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Press Syndicate of the University of Cambridge. find
  15. D’ Angelo, J. (2012). WE-informed EIL curriculum at Chuko: Towards a functional, educated, multilingual outcome. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 121–136). Bristol: Multilingual Matters. find
  16. Davidson, F. (1993). Testing English across cultures: summary and comments. World Englishes, 13(1), 113–115. ↗
  17. Davidson, F. (2006). World Englishes and test construction. In B. Kachru, Y. Kachru & C. Nelson (Eds.), The handbook of world Englishes (pp. 709–717). Malden: Blackwell Publishes Ltd. find
  18. Davies, A., Hamp-Lyons, L., & Kemp, C. (2003). Whose norms? International proficiency tests in English. World Englishes, 22(4), 571–584. find
  19. Derwing, T., & Munro, M. (1997). Accent, intelligibility and comprehensibility: evidence from four L1s. Studies in Second Language Acquisition, 19, 1–16. find
  20. Eagly, A. H., & Chaiken, S. (1993). The psychology of attitude. Fort Worth: Harcourt Brace Jovanovich. find
  21. Educational Testing Service. (2014). ETS standards for quality and fairness. Princeton: Educational Testing Service. find
  22. Elder, C., & Davies, A. (2006). Assessing English as a lingua franca. Annual Review of Applied Linguistics, 26, 282–301. ↗
  23. Elder, C., & Harding, L. (2008). Language testing and English as an international language: constraints and contributions. In Sharifian, F. and M. Clyne (eds.): Australian Review of Applied Linguistics (Special forum issue), 31(3), 34.1–34.11. ↗
  24. Fazio, R. H., Powell, M. C., & Williams, C. J. (1989). The role of attitude accessibility in the attitude-to-behavior process. Journal of Consumer Research, 16(3), 280–288. ↗
  25. Fazio, R. H., & Towles-Schwen, T. (1999). The MODE model of attitude-behavior processes. In S. Chaiken & Y. Trope (Eds.), Dual process theories in social psychology (pp. 97–116). New York: Guilford. find
  26. Field, J. (2003). Promoting perception: Lexical segmentation in L2 listening. ELT Journal, 57(4), 325–334. ↗
  27. Fulcher, G. (2003). Testing second language speaking. London: Longman. find
  28. Galloway, N., & Rose, H. (2015). Introducing Global Englishes. Abingdon: Routledge. ↗
  29. Garrett, P. (2010). Attitudes to language. key topics in sociolinguistics. Cambridge: Cambridge University Press. find
  30. Giles, H., & Billings, A. C. (2004). Assessing language attitudes: Speaker evaluation studies. In A. Davies & C. Elder (Eds.), The handbook of applied linguistics (pp. 187–209). Malden: Blackwell. ↗
  31. Gu, L., & So, Y. (2014). Voices from stakeholders: What makes an academic English test “international”? Journal of English for Academic Purposes, 18, 9–24. find
  32. Hamid, M. O. (2014). World Englishes in international proficiency tests. World Englishes, 33(2), 263–277. ↗
  33. Hamp-Lyons, L., & Zhang, B. W. (2001). World Englishes: issues in and from academic writing assessment. In L. Flowerdew & M. Peacock (Eds.), Research perspectives on English for academic purposes (pp. 101–116). Cambridge: Cambridge University Press. ↗
  34. Harding, L. (2008). The use of speakers with L2 accents in academic English listening assessment: A validation study. Unpublished doctoral dissertation, The University of Melbourne, Melbourne. find
  35. Henson, R. K. (2001). Understanding internal consistency reliability estimates: a conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177–189. ↗
  36. Hrubes, D., Ajzen, I., & Daigle, J. (2001). Predicting hunting intentions and behavior: An application of the theory of planned behavior. Leisure Sciences, 23(3), 165–178. ↗
  37. Hsu, T. H. L. (2016). Removing bias towards World Englishes: The development of a Rater Attitude Instrument using Indian English as a stimulus. Language Testing, 33 (3), 367–389. ↗
  38. Huberty, C. J., & Petoskey, M. D. (2000). Multivariate analysis of variance and covariance. In H. Tinsley & S. Brown (Eds.), Handbook of applied multivariate statistics and mathematical modeling (pp. 183–208). New York: Academic Press. ↗
  39. Jenkins, J. (2000). The phonology of English as an international language: New models, new norms, new goals. Oxford: Oxford University Press. find
  40. Jenkins, J. (2006). Current perspectives on teaching world Englishness and English as a lingua franca. TESOL Quarterly, 40(1), 157–181. find
  41. Jenkins, J. (2015). Global Englishes: a resource book for students. Abingdon: Routledge. find
  42. Kachru, B. (1992). The other tongue (2nd ed.). Urbana: University of Illinois press. find
  43. Kachru, B. (2001). World Englishes. In R. Mesthrie (Ed.), Concise encyclopedia of sociolinguistics (pp. 519–524). New York: Elsevier. find
  44. Kachru, B., Kachru, Y., & Nelson, C. (2006). The handbook of world Englishes. Oxford: Blackwell. find
  45. Kang, O. (2008). Ratings of L2 oral performance in English: Relative impact of rater characteristics and acoustic measures of accentedness. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 6, 181–205. find
  46. Kang, O., & Moran, M. (2014). Functional loads of pronunciation features in non-native speakers’ oral assessment. TESOL Quarterly, 48(1), 176–187. find
  47. Kim, H. J. (2005). World Englishes and language testing: the influence of rater variability in the assessment process of English oral proficiency. Unpublished doctoral dissertation. Iowa city: University of Iowa. find
  48. Kirkpatrick, A. (2012). English as an Asian lingua franca: the “ Lingua Franca Approach” and implications for language education policy. Journal of English as a Lingua Franca, 1(1), 121–139. find
  49. Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. The American Psychologist, 56, 16–26. ↗
  50. Kubota, R. (2001). Teaching world Englishes to native speakers of English in the USA. World Englishes, 20(1), 47–64. ↗
  51. Kunnan, A. J. (2004). Test fairness. In M. Milanvoic, C. Weir, & S. Bloton (Eds.), European year of language conference papers, Barcelona (pp. 27–48). Cambridge, U.K.: CUP. find
  52. Lee, H. (2012). World Englishes in a high school English case: A case from Japan. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 154–168). Bristol: Multilingual Matters. find
  53. Leung, C., & Lewkowicz, J. (2006). Expanding horizons and unresolved conundrums: language testing and assessment. TESOL Quarterly, 40(1), 211–234. ↗
  54. Linacre, J. M. (1989). Many-faceted Rasch measurement. Chicago: MESA. find
  55. Lindemann, S. (2002). Listening with an attitude: A model of native-speaker comprehension of non-native speakers in the United States of America. Language in Society, 31, 419–441. ↗
  56. Lowenberg, P. H. (2002). Assessing English proficiency in the expanding circle. World Englishes, 21(3), 431–435. ↗
  57. Luoma, S. (2004). Assessing speaking. New York: Cambridge University Press. find
  58. Matsuda, A., & Friedrich, P. (2011). English as an international language: a curriculum blueprint. World Englishes, 30(3), 332–344. ↗
  59. McArthur, T. (2003). The Oxford guide to world Englishes. Oxford: Oxford University Press. find
  60. McKay, S. L. (2012). Teaching materials for English as an international language. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 70–83). Bristol: Multilingual Matters. ↗
  61. McNamara, T. (1996). Measuring second language performance. London: Longman. find
  62. McNamara, T., & Roever, C. (2006). Language Testing: The Social Dimension. Oxford: Blackwell. ↗
  63. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education & Macmillan. find
  64. Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34, 520–531. ↗
  65. Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill. find
  66. Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effect size in L2 research. Language Learning, 64, 878–912. find
  67. Powers, D. E., Scheldi, M. A., Leung, S. W., & Butler, F. A. (1999). Validating the revised test of spoken English against a criterion of communicative success. TOEFL Research Report, 99(5), 63. ↗
  68. Rubin, D. L. (1992). Nonlanguage factors affecting undergraduates’ judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 33, 511–531. ↗
  69. Saito, K. & Shintani, N. (2016). Do native speakers of North American and Singapore English differentially percieve comprehensibility in second language speech? TESOL Quarterly, 50(2), 421–446. find
  70. Saito, K., Trofimovich, P., Isaacs, T., & Webb, S. (2017). Re-examining phonological and lexical correlates of second language comprehensibility: the role of rater experience. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 141–156). Bristol: Multilingual Matters. find
  71. Sewell, A. (2013). Language testing and international intelligibility: a Hong Kong case study. Language Assessment Quarterly, 10, 423–443. ↗
  72. Sharfian, F., & Marlina, R. (2012). English as an international language (EIL): An innovative academic program. In A. Matsuda (Ed.), Principles and practices of teaching English as an international language (pp. 140–153). Bristol: Multilingual Matters. find
  73. Shrout, P. E., & Yager, T. J. (1989). Reliability and validity of screening scales: Effective of reducing scale length. Journal of Clinical Epidemiology, 42, 69–78. find
  74. Smith, L. E. (1992). Spread of English and issues of intelligibility. In B. B. Kachru (Ed.), The other tongue: English across cultures (2nd ed., pp. 75–90). Urbana: University of Illinois Press. find
  75. Steenkamp, J. B. E., De Jong, M., & Baumgartner, H. (2010). Socially desirable response tendencies in survey research. Journal of Marketing Research, 47, 199–214. ↗
  76. Streiner, D. L., & Norman, G. R. (2000). Health measurement scales. A practical guide to their development and use. New York: Oxford University Press. find
  77. Taylor, L. (2002). Assessing learners’ English: but whose/which English (es)? Research Notes 10. Cambridge: University of Cambridge ESOL Examinations. find
  78. Taylor, L. (2005). Linguistic diversity: Language varieties and their implications for testing and assessment. Berlin: Paper presented at the Association of Language Testers in Europe (ALTE) Conference. find
  79. Villarreal, D., Loring, A., & Evans, K. (2014). Teaching World Englishes to undergraduates: tensions and pedagogical insights. Portland: Paper presented at American Association for Applied Linguistics Roundtable. March 22–25. find
  80. Wigglesworth, G. (2001). Influences on performance in task-based oral assessment. In M. Bygate, P. Skehan, & M. Swain (Eds.), Researching pedagogic tasks, second language learning, teaching and testing (pp. 186–209). Harlow: Longman. find
  81. Xi, X., & Mollau, P. (2011). Using raters from India to score a large-scale speaking test. Language Learning, 61(4), 1222–1255. ↗
  82. Zahn, C. J., & Hopper, R. (1985). Measuring language attitudes: the speech evaluation instrument. Journal of Language and Social Psychology, l4(2), 113–123. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hsu-2019-rater-attitude-towards-emerging-varieties/