📄 LIMSIILES: Basic English Substitution for Student Answer Assessment at SemEval 2013

Summary

The paper addresses automatic grading of short student answers in SemEval 2013 task 7 (Joint Student Response Analysis). It treats grading as paraphrase identification between a student answer and reference answers. The authors argue that meaning is easier to compare when both texts are pushed toward a small, controlled vocabulary, specifically Basic English. They mine the Simple English Wiktionary, whose definitions often pair a usage clause with an explaining clause. They run the Stanford Parser on these definitions and match up the arguments of the two clauses, producing around 20,650 phrasal paraphrase templates; one example swaps the verb "link", with arguments X and Y, for the longer expression of X making a connection to Y. These rules are applied recursively to both student and reference answers to produce simplified variants. Jaccard distance pre-ranks the variants, and lexical, WordNet and polarity features then feed an SVM. On the 5-way task the system placed third on SciEntsBank unseen questions without corpus-specific tuning. This suggests that a small collaborative dictionary can supply usable open-domain paraphrase knowledge.

Key points

• About 70% of Simple Wiktionary verb definitions consist of an independent clause plus a subordinate (often adverbial) clause. One clause shows usage and the other gives the meaning.
• Argument alignment between the "word part" and the "defining part" uses a hand-weighted score. The score combines string similarity, dependency relation, relative position and tree depth, gender/number agreement, noun–pronoun pairing, and WordNet similarity. The best complete matching is found by enumeration, or by beam search when a predicate has more than 4 arguments.
• Variant generation is time-bounded in an anytime style. It substitutes verbs and adjectives before nouns, and non-Basic words before Basic ones. For each reference answer, the n = 10 closest student variants are kept.
• Classifier features include Jaccard similarity, a polarity match based on counts of neg dependencies, the number of paraphrasing steps, WordNet similarity of nouns and main verbs, answer count, length ratio, and the number of words absent from both question and references. The classifier is a one-against-one SVM trained on Beetle and SciEntsBank.
• On the 5-way task, accuracy was 0.5330 on Beetle unseen answers and 0.4820 on SciEntsBank unseen questions. The latter was third best and above the majority (0.4110) and lexical-overlap (0.4130) baselines.

Interesting ideas

• I1. Learner dictionaries that define a word inside a full example sentence can be split into a usage clause and an explanation clause, then aligned argument by argument into reusable rewrite templates with variables.
• I2. Instead of matching two texts directly, both are normalized toward a shared restricted vocabulary, and the comparison happens in that simplified space.
• I3. Alignment of arguments between two clauses uses syntactic cues (same dependency relation, depth, position) plus agreement checks and pronoun–antecedent pairing, solved as a best complete matching with beam search for large cases.
• I4. The number of rewrite steps needed to reach a variant is kept as a feature, so distance in paraphrase space works as a confidence signal.
• I5. Polarity is approximated by counting negation dependencies in the parse and comparing the counts between the two statements. This cheaply catches contradictions that word overlap misses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): Simple Wiktionary definitions could be compiled into verb-paraphrase entries linked to VerbNet classes, so each rewrite is an explainable step in the knowledge layer.
• I2 (answer): school-test readers could check a student answer against a reference by reducing both frames to Basic-English predicates before comparing roles.
• I3 (frames): the same scored role alignment could map semantic roles between a paraphrase pair, or between a question frame and a text frame.
• I4 (learn): recording derivation depth for silver or self-induced repair rules would let training down-weight conclusions reached through long rewrite chains.
• I5 (frames): a negation-scope check on UD neg/advmod edges could flag contradictory frames before deeper reasoning.