📄 Structure-Based Evaluation of an Arabic Semantic Query Expansion Using the JIRS Passage Retrieval System

Summary

The paper tackles a classic weakness of query expansion (QE) for Arabic question answering. Adding related words to a question can retrieve more relevant passages, but it also brings in noise. The authors use a semantic QE method built on Arabic WordNet (AWN). It expands question keywords along four relation types: synonyms, definitions, subtypes (hyponyms) and supertypes (hypernyms). Their earlier evaluation of this method used keyword-based retrieval. Here they test it with JIRS, a passage retrieval system that scores passages with a distance-density n-gram model. JIRS rewards passages where the question's words appear close together and in a structure similar to the question, so noisy expansions that match only scattered words get demoted. The test set is Arabic questions taken from CLEF and TREC. The authors measure each relation type separately and the full combination against a no-expansion baseline. The structure-aware ranking lets the expanded queries improve passage quality and the number of answerable questions. This shows that richer expansion pays off when the ranker checks term structure rather than raw term overlap.

Key points

• Resource: QE draws on Arabic WordNet (AWN), using four relation types: synonyms, definitions, subtypes and supertypes.
• Retrieval system: JIRS, a language-independent passage retrieval system based on a distance-density n-gram model, replaces a plain keyword-based engine.
• Test questions: CLEF and TREC questions in Arabic.
• Experimental design:
  • Each expansion type is evaluated on its own and in combination.
  • The baseline is retrieval without QE.
  • Metrics include accuracy, MRR, number of answered questions, and the average number of keywords per question, which grows considerably after expansion.
• Main finding: the combined semantic QE gives the strongest gains over the baseline. The structure-based ranking absorbs much of the noise that expansion introduces.

Interesting ideas

• I1. Pair a recall-boosting step (expansion) with a precision-enforcing step that checks structure, so noise from the first is filtered by the second instead of being tuned away.
• I2. Evaluate each lexical relation (synonym, definition, hyponym, hypernym) separately before combining them, to see which relation helps and which mostly adds noise.
• I3. Reward a passage when query terms occur close together and in an order resembling the question's n-grams, a cheap proxy for structural match.
• I4. Track the average number of keywords per question as a diagnostic, because it shows how much an expansion inflates the query and helps explain why precision drops.
• I5. Map expansion terms through a wordnet's definitions (glosses) as well as its synsets, which can surface content words missing from synonym lists.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): when a QA reader expands a question with WordNet synonyms or hypernyms, candidate sentences should be accepted only if the expanded terms fill the same dependency/frame slots as in the question, not merely co-occur.
• I2 (learn): ablate each WordNet/FrameNet relation used in matching or frame filling, keeping a no-expansion run as a control.
• I3 (answer): use the same idea as a fallback ranking signal when a full UD parse of a long passage is unreliable, while keeping the score explainable as matched n-gram spans.
• I4 (world): log how many ontology neighbours each lookup pulls in, so explanations can show when a knowledge-graph expansion became too broad.
• I5 (world): glosses from WordNet could supply extra cue words for linking question entities to text entities, provided each link is checked against the frame structure.