📄 Evaluation of Automatically Generated Pronoun Reference Questions

Summary

The paper evaluates a system that automatically writes English pronoun reference questions: multiple-choice items in which a reader picks the antecedent of a pronoun in a passage from four options. Off-the-shelf anaphora resolution is not reliable enough to supply correct answers, so the generator works the other way round. It takes human-written Project Gutenberg texts, finds sentences with nonrestrictive relative clauses, splits each into two sentences, and swaps the relative pronoun for a personal pronoun. The relative clause's antecedent then becomes the known correct answer. Centering theory filters out splits where the new pronoun would point to a different noun. Distractors come from nearby entities, excluding anything coreferent with the answer.

The evaluation had two parts:

• Teachers rated the generated items for quality.
• University students (81 in total) answered machine-generated and human-made TOEFL-style items, and their responses were analysed with item statistics and correlations with TOEIC scores.

The machine items came out roughly on par with human ones. This suggests cheap, scalable generation of reading-comprehension test items is feasible.

Key points

• Generation pipeline: the antecedent of the relative pronoun is accepted only if two methods agree on it. One is a constituency ("lexical") parser with hand-made rules using attachment and number. The other is a dependency parser's acl:relcl relation. Sentences where they disagree are dropped.
• Teacher judgement: 39 of 60 generated questions (65%) were judged usable in a real test.
• Item discrimination: 30 machine-generated questions (MGQs) and 30 human-made questions (HMQs) from TOEFL preparation books were compared. Average discrimination was 0.33 for MGQs and 0.37 for HMQs. Using the ≥0.2 acceptability threshold, 22 MGQs and 24 HMQs passed. High and low groups were the top and bottom 27% of test takers.
• Correlation with TOEIC: MGQ scores correlated with TOEIC Reading at 0.65, Listening at 0.56 and the total at 0.74. The corresponding HMQ figures were 0.68, 0.57 and 0.77. MGQ and HMQ scores correlated with each other at 0.61.
• Difficulty: passage difficulty, measured with the JACET8000 word list, correlated 0.56 with MGQ item difficulty. In a group of 25 students, the hardest item was solved by 5 and the easiest by 24.

Interesting ideas

• I1. Turning nonrestrictive relative clauses into split sentences with a personal pronoun yields pronoun–antecedent pairs whose answer is known by construction, which is a cheap way to manufacture reliable coreference examples.
• I2. An example is kept only when two independent analyses (rule-based constituency traversal and a dependency parser) agree, and discordant cases are simply thrown away rather than adjudicated.
• I3. After rewriting text, the authors re-check with Centering theory that the newly introduced pronoun would still be read as pointing to the intended entity.
• I4. Distractors are drawn by recency, entities coreferent with the answer are excluded, and the error analysis showed that a distractor mentioned twice lured even strong readers.
• I5. Item quality is measured with a discrimination index between top and bottom 27% groups, and test scores are validated against an external proficiency measure.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): we could auto-build a coreference test and training set from any parsed English corpus, using UD acl:relcl arcs as the source of truth for the pronoun's antecedent.
• I2 (learn): this matches our witness-parser silver data, and supports discarding disagreements outright instead of forcing a vote when precision matters more than coverage.
• I3 (back): when generating text from the meaning graph, we can run a salience-based check (subject outranks a PP object) on each emitted pronoun and fall back to a full noun phrase if the likely reading would be wrong.
• I4 (answer): when Mova builds or checks multiple-choice items for fables and school tests, it should exclude coreferent options and treat repeated mentions as a known trap to test its own resolver against.
• I5 (learn): we could score our evaluation questions by whether they separate stronger from weaker model versions, and drop items that every version passes or fails.