📄 QU-BIGIR at SemEval 2017 Task 3: Using Similarity Features for Arabic Community Question Answering Forums

Summary

The paper addresses SemEval 2017 Task 3, Subtask D, which concerns Arabic community question answering. A search engine retrieves 30 earlier question-answer pairs for each new question. The system must re-rank these pairs by relevance and label each one as relevant or irrelevant.

The Qatar University team, QU-BIGIR, frames this as supervised learning. A linear SVM is trained over a small set of pairwise similarity features. The lexical features are:

• synonym-aware overlap using Arabic WordNet,
• a Dirichlet-smoothed language-model score,
• TF-IDF cosine,
• three Jaccard variants.

The semantic features compare texts through the average of their word2vec vectors and through the covariance matrix of those vectors. Every feature is computed three ways: the new question against the old question, against the answer, and against both together. The SVM decision value serves as the ranking score, and its sign serves as the label.

The best run combined lexical and embedding features and reached MAP 59.48, placing second in the subtask. This shows that cheap, transparent similarity signals can compete in ranking tasks with limited resources.

Key points

• Each new question comes with 30 candidate QApairs, and the system must both rank them and classify them as true or false.
• The feature set grows across runs:
  • Contrastive-1 uses 6 term-based features for each of 3 setups (QQ, QA, QQA), giving 18 features.
  • Contrastive-2 adds average-embedding cosine and covariance-embedding cosine, giving 24 features.
  • Primary additionally fuses scores with a separate SVM trained on covariance vectors.
• Background statistics come from ArabicWeb16, and the LM score uses µ = 2000, which is Lucene's default.
• The word2vec model was trained with Gensim (d = 100) on the task's own Arabic data, with test data held out.
• Official test MAP was 59.48 for Contrastive-2, 59.13 for Contrastive-1 and 56.69 for Primary. The IR baseline scored 60.55, though it does no labeling. Contrastive-2 also had the best accuracy (66.15).
• On the development set the Primary run was best (MAP 43.41 vs 42.87 and 42.54). The authors attribute its weaker test result to the test set containing only medical questions.

Interesting ideas

• I1. A text can be summarized by the covariance of its word vectors, not only by their mean, which gives a fixed-length second-order descriptor that is cheap to compute.
• I2. The same similarity can be measured against three targets (the old question, the answer, and both joined) to reveal where the match actually lives.
• I3. Asymmetric Jaccard variants divide the overlap by the size of one side only, turning similarity into directional coverage.
• I4. Overlap counts a term as matched if the other text has it or a WordNet synonym of it, which is a simple and inspectable form of soft matching.
• I5. A configuration that won on the development set lost on a test set that had shifted toward medical topics, which warns against tuning fusion weights on one domain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova could use it as a strong bag-of-vectors negative control, so that structure-based answering must demonstrably beat it on the same items.
• I2 (answer): when picking supporting passages for a fable question, Mova could score the question frame separately against the passage's events and against its stated outcomes, then report which one matched.
• I3 (frames): Mova could measure what share of a question's frame slots are filled by a candidate frame, independently of how much extra the candidate contains.
• I4 (world): when aligning question entities and predicates to the text's world, Mova could accept a lemma match through a shared WordNet synset and record that synset as the explanation.
• I5 (learn): Mova's evaluation could hold out an entire genre, such as tales versus school tests, before accepting any tuned combination of components.