📄 A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation

Summary

Large vision-language models handle single-page documents well, but they struggle when an answer depends on content spread across many pages. Arabic makes this worse because existing benchmarks such as Camel and KITAB cover only short, single-page material. The authors build two automated components. The first is an agentic pipeline that crawls, filters and normalizes long Arabic documents from domains such as education, finance, government reports, news and technical manuals. The second is a closed-loop question-answer generator. In it, a question-generator LVLM writes context-aware questions, a swarm of answer-generator LVLMs tries to solve them, and an evaluator scores the results. Low-confidence outputs are sent back for regeneration. The quality threshold is a tunable hyperparameter, so the difficulty of the questions can be dialed up or down. The output is AraLongBench, a benchmark of single- and cross-page questions over documents spanning hundreds of pages. Under stricter difficulty gates, leading LVLMs lose roughly 20 accuracy points, which exposes weak long-context understanding of Arabic.

Key points

• Collection pipeline tools:
  • BeautifulSoup scraping for HTML, pdfplumber for native PDFs, and Tesseract OCR with binarization and deskewing for scans.
  • A handler based on the Unicode Bidirectional Algorithm fixes right-to-left misalignment.
• Preprocessing: pages are rasterized with pdf2image, segmented with DocLayout-YOLO into headings, paragraphs, tables and figures, and then split into overlapping chunks.
• Benchmark buckets:
  • By length: short (<100 pages), medium (100–200 pages) and long (>200 pages).
  • By evidence location: single-page (SP) and cross-page (CP) questions.
• Evaluation: models were tested zero-shot under three gates (No Gate, 50%, 25%).
  • Closed models: GPT-4o, Gemini-2.0 Flash, Gemini-1.5 Pro and Gemini-2.5 Pro.
  • Open models: AIN 7B, Aya Vision 32B, Qwen 2 VL 72B and Qwen 2.5 VL 72B.
  • Example drops: GPT-4o SC went from 87.2% to 65.7%, Gemini-2.0 Flash SP from 94.3% to 73.9%, and AIN SC from 78.5% to 58.2%.
• Evidence-source check: without this component, a manual audit of 100 sampled questions found a 14% mismatch between the cited evidence and the answer. With the component, the mismatch fell below 5%.

Interesting ideas

• I1. A crowd of different answer models acts as a difficulty meter: a question is kept or rejected based on how many of them solve it, so no human judge is needed. — For Mova (learn): Mova could generate questions about a text and keep those on which several witness parsers or reader variants disagree, using them as targeted training and repair material.
• I2. Treating the acceptance threshold as an explicit knob produces graded test sets, and one model can be reported across several gates instead of with a single score. — For Mova (answer): school-test and fable-QA results could be reported at several difficulty tiers, showing where reasoning breaks down rather than hiding it in an average.
• I3. Each generated answer must name its evidence source, and a small random audit measures how often the citation and the answer actually agree. — For Mova (answer): Mova's readers could attach the supporting sentence or world-graph node to every answer, and a periodic 100-sample audit could track the mismatch rate as a negative control.
• I4. Questions are bucketed both by document length and by whether the evidence sits in one place or is spread out, which separates local reading from long-range linking. — For Mova (world): QA tests could be split into same-sentence and cross-paragraph evidence, isolating failures in coreference and event ordering inside the text world.
• I5. Low-confidence outputs automatically trigger regeneration, so the generator-solver-judge loop keeps hardening its own data without supervision. — For Mova (learn): a self-play loop could let Mova pose questions about a text, answer them, and recycle the ones where confidence is low into new repair rules or teacher-tier queries.