📄 Semantic Equivalence Detection: Are Interrogatives Harder than Declaratives?

Summary

The paper asks whether detecting semantic equivalence is harder for questions than for statements. Paraphrase detection between declaratives and Duplicate Question Detection (DQD) are both forms of Semantic Text Similarity (STS), but the authors note that synonymy works differently for each. Two statements are equivalent when swapping them keeps truth conditions intact. Two questions are equivalent when every good answer to one also answers the other. To compare the cases fairly, they build equal-sized, class-balanced subsets from four English corpora: MSRPC (declaratives), Quora (interrogatives), DupStack (mixed), and PIT (tweets). They then run the established DQD approaches from their earlier work on each subset, with DCNN emerging as the strongest. Three findings stand out:

• Duplicate question pairs share less surface material than paraphrase pairs, as measured with BLEU.
• Resolvers score clearly lower on questions than on statements.
• Tweets are the hardest segment type of all.

They also show that adding cheaply harvested question pairs to declarative training data improves declarative paraphrase detection, but only if the original declarative data stays in the mix. This matters when comparing scores across datasets of different segment types.

Key points

• Each of the four corpora (MSRPC, Quora, DupStack, PIT) contributes 7,800 pairs, split evenly into 3,900 duplicates and 3,900 non-duplicates. The data is divided 80/20 into training and test sets.

• MSRPC has only 1,901 genuine negatives. The authors created the remaining 1,999 by randomly pairing sentences drawn from different pairs.

• In cross-corpus experiments, DCNN performs best overall. Interrogatives prove harder than declaratives, and tweets are the hardest.

• Results on MSRPC test data (accuracy) by training set:

  Training data	Accuracy
  MSRPC alone	78.42
  Quora100k alone	74.42
  MSRPC + Quora100k	79.36
  All four sets merged	79.42

• Merging datasets helps resolvers for declaratives but not resolvers for interrogatives.

Interesting ideas

• I1. Equivalence of statements and equivalence of questions need different tests: substitution that preserves truth for statements, interchangeable answers for questions.
• I2. Before claiming one input type is harder than another, the authors equalize dataset size and class balance across corpora, so differences reflect the input type rather than the data conditions.
• I3. When natural negative examples are scarce, extra negatives can be made by recombining sentences from unrelated pairs.
• I4. Measuring surface overlap (BLEU) between the two members of a pair is a quick diagnostic of how much a task rewards shallow matching.
• I5. A small hand-annotated set can be enlarged with a large free byproduct dataset of a related type, but only while the in-type data stays in the mix; the out-of-type data alone performed worse.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Two user questions could be treated as the same when they resolve to the same answer set in the world of the text, instead of comparing their wording.
• I2 (learn): Evaluations that compare genres (fables, school tests, word problems) should use matched-size, balanced samples so the scores are actually comparable.
• I3 (learn): This is a cheap way to generate negative controls for frame or coreference matchers, though such easy negatives should be reported separately from hard ones.
• I4 (back): The same overlap measure could check that text → graph → text paraphrases change the wording while keeping the meaning, so the system is not just copying the input.
• I5 (learn): When adding silver or out-of-genre data to gold treebanks or frame annotations, the gold data should always be kept in training, and a gold-only baseline should be tracked.