📄 UPC-USMBA at SemEval-2017 Task 3: Combining multiple approaches for CQA for Arabic

Summary

The UPC-USMBA team took part in SemEval-2017 Task 3, subtask D (Arabic community question answering). In this subtask, a system receives a new medical question together with 30 question–answer pairs taken from forum threads, and must reorder those pairs by how relevant they are to the new question. The previous year's system relied on only two shallow-feature classifiers and performed poorly. This time the team built several independent "atomic" scorers and merged their outputs:

• lexical string-based classifiers for Arabic and for English,
• a Lucene retrieval baseline,
• LSI and LDA topic models trained on medical Arabic web corpora,
• several word and document embeddings,
• hand-written question-type rules.

Every Arabic text was also machine-translated into English, so that English tools could be applied to it. A bilingual medical terminology supplied domain entities. The team searched over subsets of the atomic scorers and over a small grid of fusion settings. The best combination, LSI plus LDA trained on lemmatized Webteb text, became the primary run. It placed 2nd of 3 teams by MAP and 1st by accuracy. The work shows that a mix of cheap, diverse scorers can be competitive in a specialised Arabic domain.

Key points

• Task setup: Subtask D asks systems to rerank 30 question–answer pairs for each original Arabic question. MAP is the official measure, and the team also tracked accuracy.
• Preprocessing pipeline:
  • Google Translate API produced English versions of all texts.
  • English was processed with Stanford CoreNLP and Arabic with Madamira.
  • Results were enriched with WordNet synsets.
  • A medical terminology with three categories (body parts, drugs, diseases) supplied named entities.
• Atomic scorers:
  • Basic_ar and Basic_en used Weka logistic regression over shallow, vectorial and domain features.
  • LSI and LDA came from gensim, trained on Wikipedia, Webteb, altibbi and dailymedical-info.
  • Embeddings tried were Word2Vec, GloVe and doc2vec. Doc2vec was the strongest of these but still lost to LSI+LDA.
  • The rule system had 75 rules covering 13 question types for Arabic and 16 for English.
• Fusion search:
  • The team tried the whole powerset of atomic scorers. Combinations of more than three components did not give good results.
  • Score merging used max or average.
  • Thresholding was none, global or local, at levels 0.2, 0.4, 0.6 or 0.8.
  • The final decision used max, voting or coincidence.
• Official results: The primary run scored MAP 0.5773 and accuracy 0.6624. The winner, GW QA, had MAP 0.6116. On the test set, the Lucene scorer alone reached 58.32 MAP, which was better than the submitted combination.

Interesting ideas

• I1. Rules can be checked across languages: when a rule fires on a pair in Arabic, its counterpart should usually fire on the English translation, so disagreements reveal rules that need repair or are missing.
• I2. Questions are classified into typed templates with filled slots (for example CauseDisease with a Disease slot), and a question pair counts as matching only when the types agree and the slot values are compatible; an answer rule that confirms the slot then raises confidence further.
• I3. Because their scorers were not true rankers, the authors reported per-item accuracy alongside the ranking metric, which separates errors in the relevance decision from errors in the ordering.
• I4. An exhaustive search over subsets of components, combined with a small grid of fusion settings, showed that adding more components quickly stopped helping, and a plain Lucene baseline still beat the chosen mix on test data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Disagreement between parallel rule sets, or between parsers working on paraphrases, could be logged automatically and used to propose candidates for self-induced repair rules.
• I2 (answer): Readers could assign each question a typed template with slots and then check that the candidate answer actually fills that slot, which keeps every match decision explainable.
• I3 (answer): Test suites could score the yes/no relevance or correctness decision apart from how answers are ranked, so it is clear which part of the pipeline is failing.
• I4 (learn): Every ensemble of witnesses should be compared against the simplest single baseline on held-out data, which works as a negative control against overfitting the combination.