Elamur Research⌕
☰
home / papers / 2022 / SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners

SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners

Paper · 2022 · Qiongqiong Liu, Yaying Huang, Zitao Liu et al.

Summary

Sentence completion (SC) items give a sentence with one or more gaps and three to five candidate fillers. They are a staple of English as a Second Language (ESL) exams. Existing public SC collections, such as the MSR SC dataset, hold only a few thousand items and use a fixed gap and option format. That makes them too small and too uniform for modern pre-trained models. The authors release SC-Ques, which contains 289,148 cleaned SC questions taken from real K-12 English assignments written by teaching staff at a large Chinese edtech company. Each question is labelled by gap count and option length.

They benchmark large versions of BERT, RoBERTa, XLNet, BART and DeBERTaV3. Each option is inserted into the stem, and the resulting full sentence is classified as right or wrong. DeBERTaV3 gives the best results across stem and option lengths.

A confidence-threshold analysis shows that a solver can decline uncertain items and stay highly precise while still answering most questions. The resource matters for automated tutoring feedback, distractor quality checks and personalized practice.

Key points

  • Dataset size and split: SC-Ques has 289,148 questions with 3 or 4 options each, after removing duplicates and incomplete items. It is split roughly 5:1 into 241,195 training and 47,953 test questions.
  • Four categories by gap count and longest-option length:
    • C1, one gap and one token: 110,645 questions.
    • C2, one gap and several tokens: 133,249 questions.
    • C3, several gaps and one token: 27,886 questions.
    • C4, several gaps and several tokens: 17,368 questions.
  • Short contexts: 84.35% of questions have a single gap. Most stems are 5–15 words long, with 9 words the most common length, and only 5.88% exceed 20 words.
  • Benchmark design: each filled-in sentence is encoded by the LM, and a two-layer MLP head makes a binary right/wrong decision trained with cross-entropy. Accuracy is the metric.
    • DeBERTaV3 leads every bucket, for example 0.9543 on one-gap items whose longest option has 2 words.
    • Accuracy dips slightly for very short and very long options.
  • Abstention results: option scores are sharpened with a temperature-0.1 softmax and then thresholded.
    • At a threshold of 0.95, most models exceed 97% precision while still answering over 80% of questions.
    • DeBERTaV3 stays above 95% precision and 87% coverage at every threshold.
  • Availability: data and code are on GitHub at ai4ed/SC-Ques.

Interesting ideas

  • I1. Turn a multiple-choice gap item into several complete candidate sentences and judge each one on its own, so the solver only has to rate whole sentences as acceptable.
  • I2. Let the solver abstain when its top option is not clearly ahead of the others, and report precision against coverage rather than accuracy alone, since a confident wrong answer misleads learners.
  • I3. Break evaluation down by structural traits of each item (number of gaps, longest option length, stem length) to see where a solver weakens.
  • I4. Exam distractors are near-miss variants of a correct sentence, often differing only in an inflection or function word, which makes them natural minimal-pair negatives.
  • I5. A solver that scores all options can also audit the options themselves, flagging distractors that are too easy or arguably correct so item writers can revise them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): School-test readers could fill each option in, parse the result, and choose the candidate whose UD tree passes agreement and valency checks, giving an explainable reason for the rejection.
  • I2 (answer): Students answering tests could output "not sure" below a margin, and evaluation could report a precision-coverage curve instead of a single score.
  • I3 (learn): Evaluation could be sliced the same way, for example by sentence length or construction type, so a regression in one slice is not hidden by the overall average.
  • I4 (tokens): The wrong options could serve as negative controls for the tagger and morphology, checking that pairs like is/are or much/many produce feature mismatches.
  • I5 (frames): Pragmatics could flag options that are grammatical but only pragmatically odd, separating syntax-based distractors from meaning-based ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Jinan University.

Cite

Cite as Qiongqiong Liu, Yaying Huang, Zitao Liu, Shuyan Huang, Jiahao Chen, Xiangyu Zhao, Guimin Lin, Yuyu Zhou and Weiqi Luo. 2022. SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners. arXiv:2206.12036.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.12036 · arXiv 2206.12036

@misc{liu2022sc,
  title = {{SC-Ques: A Sentence Completion Question Dataset for English as a Second Language Learners}},
  author = {Qiongqiong Liu, Yaying Huang, Zitao Liu, Shuyan Huang, Jiahao Chen, Xiangyu Zhao, Guimin Lin, Yuyu Zhou and Weiqi Luo},
  year = {2022},
  doi = {10.48550/arxiv.2206.12036},
  eprint = {2206.12036},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.12036}
}

References

6 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. Argouarc’h, J.: Dependency, skip-grams, association tensors and machine learning for sentence completion. In: Proc. 14th Conf. Natural Lang. Process (2018) 2. Banerjee, S., Bhaskar, P., Pakray, P., Bandyopadhyay, S., Gelbukh, A.: Multiple choice question (mcq) answering system for entrance examination. In: Cross-Language Evaluation Forum (2013) 3. Beinborn, L., Zesch, T., Gurevych, I.: Candidate evaluation strategies for improved difficulty prediction of language tests. In: Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (2015) 4. Beinborn, L.M.: Predicting and Manipulating the Difficulty of Text-completion Exercises for Language Learning. Ph.D. thesis, Technische Universität Darmstadt (2016) 5. Chen, J., Liu, Z., Huang, S., Liu, Q., Luo, W.: Improving interpretability of deep sequential knowledge tracing models with question-centric cognitive representations. AAAI Conference on Artificial Intelligence (2023) find
  2. Title Suppressed Due to Excessive Length 6. Chen, M., D’Arcy, M., Liu, A., Fernandez, J., Downey, D.: CODAH: An adversarially-authored question answering dataset for common sense. In: Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP (2019) 7. Chen, S.F.: Performance prediction for exponential language models. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (2009) 8. Chen, S.F.: Shrinking exponential language models. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (2009) 9. Chesani, F., Mello, P., Milano, M.: Solving mathematical puzzles: A challenging competition for AI. AI Magazine 38(3) (2017) 10. Davey, G., De Lian, C., Higgins, L.: The university entrance examination system in china. Journal of Further and Higher Education 31(4) (2007) 11. Donahue, C., Lee, M., Liang, P.: Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020) 12. Fedus, W., Goodfellow, I., Dai, A.M.: MaskGAN: Better text generation via filling in the . In: International Conference on Learning Representations (2018) 13. Franke, W.: The reform and abolition of the traditional Chinese examination system, vol. 10.
  3. Harvard Univ Asia Center (1960) 14. Graves, A., Mohamed, A.r., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (2013) 15. Gubbins, J., Vlachos, A.: Dependency language models for sentence completion. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (2013) 16. He, P., Gao, J., Chen, W.: DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In: The Eleventh International Conference on Learning Representations (2023) 17. He, P., Liu, X., Gao, J., Chen, W.: DeBERTa: Decoding-enhanced bert with disentangled attention. In: International Conference on Learning Representations (2021) 18. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2018) 19. Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (2019) 20. Khot, T., Sabharwal, A., Clark, P.: Scitail: A textual entailment dataset from science question answering. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018) 21. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 22. Lee, K., Lee, G.G.: Sentence completion task using web-scale data. In: 2014 International Conference on Big Data and Smart Computing (2014) 23. Lev, I.: Logic puzzles: A new test-suite for compositional semantics and reasoning (2006) 24. Lev, I., MacCartney, B., Manning, C.D., Levy, R.: Solving logic puzzles: From robust processing to precise semantics. In: Proceedings of the 2nd Workshop on Text Meaning and Interpretation (2004) ↗
  4. Q. Liu et al. 25. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020) 26. Liu, D., Fu, J., Liu, P., Lv, J.: TIGS: An inference algorithm for text infilling with gradient search. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019) 27. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019) 28. Liu, Z., Liu, Q., Chen, J., Huang, S., Gao, B., Luo, W., Weng, J.: Enhancing deep knowledge tracing with auxiliary tasks. The International Conference of World Wide Web (2023) 29. Liu, Z., Liu, Q., Chen, J., Huang, S., Luo, W.: simpleKT: A simple but tough-to-beat baseline for knowledge tracing. International Conference on Learning Representations (2023) 30. Liu, Z., Liu, Q., Chen, J., Huang, S., Tang, J., Luo, W.: pyKT: A python library to benchmark deep learning based knowledge tracing models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022) 31. Madaus, G.F.: The effects of important tests on students: Implications for a national examination system. The Phi Delta Kappan 73(3) (1991) 32. Mikolov, T., Kombrink, S., Burget, L., Černockỳ, J., Khudanpur, S.: Extensions of recurrent neural network language model. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (2011) 33. Mohammad, S., Dorr, B., Hirst, G.: Computing word-pair antonymy. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (2008) 34. Ng, H.T., Teo, L.H., Kwan, J.L.P.: A machine learning approach to answering questions for reading comprehension tests. In: 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (2000) 35. Nozza, D., Bianchi, F., Hovy, D.: HONEST: Measuring hurtful sentence completion in language models. In: The 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (2021) 36. Park, H., Park, J.: Assessment of word-level neural language models for sentence completion. Applied Sciences 10(4) (2020) 37. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (2018) 38. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8) (2019) 39. Riloff, E., Thelen, M.: A rule-based question answering system for reading comprehension tests. In: ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems (2000) 40. Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45(11) (1997) ↗
  5. Title Suppressed Due to Excessive Length 41. Shen, T., Quach, V., Barzilay, R., Jaakkola, T.: Blank language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (2020) 42. Tang, E.: Assessing the effectiveness of corpus-based methods in solving SAT sentence completion questions. Journal of Computers 11(4) (2016) 43. Tran, K.M., Bisazza, A., Monz, C.: Recurrent memory networks for language modeling. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016) 44. Turney, P.: A uniform approach to analogies, synonyms, antonyms, and associations. In: Proceedings of the 22nd International Conference on Computational Linguistics. find
  6. Manchester, UK (2008) 45. Turney, P.D., Littman, M.L.: Corpus-based learning of analogies and semantic relations. Machine Learning 60(1-3) (2005) 46. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in Neural Information Processing Systems 30 (2017) 47. Wang, D., Boytsov, L., Araki, J., Patel, A., Gee, J., Liu, Z., Nyberg, E., Mitamura, T.: CMU multiple-choice question answering system at NTCIR-11 QA-Lab. In: Proceedings of the 11th NTCIR Conference on Evaluation of Information Access Technologies (2014) 48. Woods, A.: Exploiting linguistic features for sentence completion. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (2016) 49. Yang, T., Deng, H.: Intelligent sentence completion based on global context dependent recurrent neural network LM. In: Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing (2019) 50. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32 (2019) 51. Zhu, W., Hu, Z., Xing, E.: Text infilling. arXiv preprint arXiv:1901.00158 (2019) 52. Zweig, G., Burges, C.J.: The microsoft research sentence completion challenge. Tech. rep., Citeseer (2011) 53. Zweig, G., Platt, J.C., Meek, C., Burges, C.J., Yessenalina, A., Liu, Q.: Computational approaches to sentence completion. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2012) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2022-sc-ques-sentence-completion-question/