📄 QU-IR at SemEval 2016 Task 3: Learning to Rank on Arabic Community Question Answering Forums with Word Embedding

Summary

This system paper addresses answer re-ranking for Arabic community question answering in SemEval 2016 Task 3, Subtask D. For each newly posted question, a search engine returns 30 earlier question–answer pairs, and the task is to order them by relevance. The data were collected from Arabic medical forums (webteb, altibbi) and Consult Islamweb.

The authors pair the new question with each retrieved candidate to form a document. They train a word2vec model with Gensim on the training collection and describe each document by a dense vector. That vector is either the mean of its word embeddings or the flattened triangular half of the covariance matrix of those embeddings. Pointwise learning-to-rank models from RankLib (MART and Random Forests) then score the candidates.

The primary run reached MAP 38.63, placing fourth among primary entries and 29.7% above the 29.79 baseline. After the deadline, covariance features raised MAP to 41.07, a 37.9% gain. The work suggests that second-order statistics of embeddings hold relevance signal that plain averaging misses, at very low engineering cost.

Key points

• The training data contain 1,031 questions with 30,411 candidate QApairs labelled Direct, Relevant or Irrelevant (about 3% / 57% / 40%). Development data contain 7,355 question–QApair items, after one malformed question was dropped.
• Three document setups were compared:
  • QQA: new question plus the full candidate pair.
  • QA: new question plus the answer only.
  • QQ: new question plus the retrieved question only.
• QQA worked better than QA. The contrastive-1 run used QA with Random Forests and scored the lowest MAP, 37.80.
• Pointwise L2Rank methods beat the pairwise and listwise ones the authors tried. The official runs scored 38.63 (MART, QQA), 37.80 and 39.07 (Random Forests, QQA, 80/20 train/validation split).
• CovWE-50 reached 41.07 and CovWE-100 reached 40.68. Training took about 44.5 minutes for CovWE-50 versus 5.27 hours for CovWE-100.
• Adding tf-idf vectors over the 5000 most frequent unigrams did not help. Feature normalization (z-score or linear) tended to raise MAP.
• To produce binary labels, the 10 top-ranked candidates were marked true and the rest false.

Interesting ideas

• I1. Pooling a text's word vectors through their covariance captures how dimensions vary together, not just their centre, and it beat the mean in this setting.
• I2. Systematically varying which parts of a candidate are fed in (QQA vs QA vs QQ) reveals where the useful matching signal lives.
• I3. With modest labelled data, a simple per-item (pointwise) scorer outperformed pairwise and listwise rankers.
• I4. Halving the embedding size kept accuracy while cutting training time several-fold, so dimensionality should be tested rather than assumed.
• I5. Turning a ranking into true/false labels with a fixed top-k cutoff is crude but transparent; the authors note a score threshold or a classifier as alternatives.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): such an orderless covariance descriptor makes a strong negative-control baseline, showing how far a structure-free reader gets before crediting the UD-tree-and-frames pipeline with any gain.
• I2 (answer): when choosing supporting sentences for fable or school-test QA, compare matching against the question part versus the answer part to learn which evidence source the reader should weight.
• I3 (answer): ranking candidate answers for test questions could start from an explainable per-candidate score before trying more complex comparative ranking.
• I4 (tree): before scaling gparse's hidden or embedding sizes, run a smaller-dimension variant to check whether the extra cost actually buys LAS.
• I5 (answer): when ranked options must become yes/no verdicts, an explicit, logged cutoff rule keeps the decision explainable and easy to audit.