📄 Parameter Efficient Fine Tuning Llama 3.1 for Answering Arabic Legal Questions: A Case Study on Jordanian Laws

Summary

The paper looks at how well a mid-sized open LLM can answer questions about Jordanian legislation in Arabic. This is a low-resource setting: there is little annotated legal data, and Arabic has complex grammar. The authors gathered 18 Jordanian laws and split them into individual articles. They then used GPT-3.5 Turbo to turn each article into (question, answer, context) triples, which were checked by hand, for roughly 6,000 pairs in total. Two 4-bit quantized Llama 3.1 8B checkpoints, the base model and the Instruct model, were adapted with LoRA through the Unsloth framework on a single Colab A100. In both training and testing, the model sees the relevant article as context, which imitates the reading step of a RAG pipeline. BLEU and ROUGE were measured against each model's own untuned version. BLEU rose substantially for both models. ROUGE improved only for the base model. The work is presented as groundwork for offline Arabic legal RAG systems, and the code and data are public.

Key points

• The source corpus is 18 Jordanian laws, including the Civil, Penal, Labor, Companies, Electronic Crimes 2024 and Arbitration laws, broken into 3,578 articles.
• GPT-3.5 Turbo produced about 6,000 QA pairs with context, which were manually reviewed and split 80/20 into train and test sets.
• The models were Llama-3.1-8B-bnb-4bit and Llama-3.1-8B-Instruct-bnb-4bit. Training used LoRA (r=16, lora_alpha=16, dropout 0), learning rate 2e-4, batch size 2 with gradient accumulation, and a single epoch. Only about 6 GB of the A100's 40 GB was reserved.
• Base model: BLEU went from 0.058 to 0.290 and ROUGE-1 from 0.026 to 0.081.
• Instruct model: BLEU went from 0.128 to 0.270, but ROUGE-1 fell from 0.070 to 0.063 and ROUGE-2 from 0.013 to 0.005. This contradicts the paper's statement that both metrics improved.
• After tuning, the plain base model came out slightly ahead of the tuned Instruct model.
• Code and data are available at github.com/msfasha/Research-Resources (ArabicLegalLLM).

Interesting ideas

• I1. Cheap domain QA data comes from treating each statute article as a self-contained unit and asking a stronger LLM to write a question, answer and supporting context for it, followed by a human pass. Data generation stays tied to small, traceable source passages. — For Mova (learn): an LLM teacher tier could generate QA triples per fable paragraph, with the paragraph id attached, so every silver answer can be traced to the exact span it came from.
• I2. Training and testing both supply the gold passage as context, so the model learns to read the given text rather than recall memorized facts. This cleanly separates the reading skill from the retrieval skill. — For Mova (answer): fable and tale QA could be benchmarked in a "passage given" mode, so failures point to the reader (frames → world) rather than to finding the right sentence.
• I3. Each tuned model is scored against its own untuned twin. The resulting table also shows BLEU and ROUGE disagreeing for the Instruct model, a reminder that overlap metrics alone can hide regressions. — For Mova (learn): every training run could report the delta against its own starting checkpoint on several metrics, flagging any metric that drops even when the headline one rises.
• I4. The authors suggest that a neutral base checkpoint adapted better than an instruction-tuned one, perhaps because it carries fewer prior habits to unlearn. — For Mova (tree): when fine-tuning gparse on a new treebank or UD dialect, it may be worth comparing a start from a general checkpoint against one already specialised to another dialect.