📄 MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language

Summary

Large language models fall out of date, and knowledge editing tries to inject new facts without retraining. Multi-hop questions are a strict test because one changed fact must propagate through a chain of reasoning, the "ripple effect." Prior work on this, MQA-KE, has targeted English almost exclusively. This paper brings the task to Arabic with MQA-KEAL. The system stores each edit as a subject–relation–object triplet, indexed with Contriever, instead of as free text. An LLM breaks the question into sub-questions. The system then answers each sub-question in turn from the edit memory or the target LLM, and logic rules (implication and compositional) prune the candidate answers. The authors also release two benchmarks:

• MQuAKE-AR, a human-validated Arabic translation of MQuAKE.
• MQA-AEVAL, a new set of questions about the Arabian Peninsula.

Across GPT-3.5-turbo-instruct, AceGPT-13B and Jias-13B, MQA-KEAL beats FT, ROME, MEMIT and MeLLo by a clear margin, and it degrades less as the number of edits grows. This shows that structured edit memory scales better than embedding raw sentences.

Key points

• Benchmarks:
  • MQuAKE-AR comes in T and CF variants. MQuAKE-T-AR is tested with 1, 100 and 1868 edits.
  • MQA-AEVAL is tested with 1, 100 and 229 edits and mostly covers recent events after the models' training cut-offs.
• Best result: GPT-3.5-turbo-instruct on MQuAKE-T-AR scores M-Acc 79.52 / H-Acc 73.46 with 1 edit, falling to 65.45 / 57.33 with 1868 edits.
• Ablation of the rules:
  • Removing implication rules lowers M-Acc by about 9–11%.
  • Removing compositional rules costs only 0.8–4.3%.
  • With both removed, the model still reaches 55.01 M-Acc at 1868 edits, versus 45.41 for MeLLo.
• Model ranking: GPT-3.5 does best, then AceGPT-13B, then Jias-13B. The authors credit AceGPT's Llama-2 instruction tuning for better question decomposition.
• Error analysis (50 sampled failures):
  • 25% come from structured retrieval.
  • 17% come from bad decomposition.
  • 11% come from the target LLM's own wrong answers.
  • The rest are miscellaneous.
  • Missing entity aliases in Arabic is named as a hurdle.

Interesting ideas

• I1. Storing updates as relation triplets rather than embedded sentences lets the system filter retrieved facts by relation type, which keeps similar-looking edits apart as memory grows.
• I2. A multi-hop question is split into a chain of single-relation sub-questions, and each intermediate answer becomes the subject of the next hop.
• I3. Candidate answers are generated generously and then pruned by implication and compositional logic rules; the broader-coverage implication rules contribute far more.
• I4. The ripple-effect test checks whether one changed fact correctly alters every downstream answer, not just the directly edited one.
• I5. Error analysis attributes each failure to one pipeline stage (retrieval, decomposition, generator).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): New facts asserted in a text could enter the world model as typed relation tuples keyed by relation, so later lookups filter by relation instead of by fuzzy similarity.
• I2 (answer): Fable and school-test readers could answer compositional questions as explicit hop chains over the world graph, with every hop logged as an explanation step.
• I3 (answer): After a reader proposes several answers, consistency rules drawn from the ontology could reject candidates that contradict known relations, and per-rule ablation would show which rules actually help.
• I4 (learn): Evaluation could include perturbation controls where a single fact in a tale is altered, and the test checks that every dependent answer flips while unrelated answers stay fixed.
• I5 (learn): Sampling failures and assigning each to tokens, tree, frames, world or answer would show which stage to repair first.