DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning

Paper · 2026 · Ruiyao Xu, Noelle I. Samia and Han Liu

Summary

Instruction-tuning an LLM for a specialized field usually requires expensive expert annotation. The alternatives are domain corpora, which are often proprietary or missing, or human-written seed examples. DS2-Instruct removes these requirements. It is a zero-shot pipeline that builds domain instruction data from only a task definition.

The pipeline has three stages:

Keyword pool. It grows a pool of domain keywords by repeatedly asking an LLM for both more basic prerequisite concepts and more advanced ones. It then adds terms mined from passages retrieved out of a mixed document collection.
Instruction generation. It combines these keywords with the cognitive levels of Bloom's Taxonomy to vary question types.
Filtering. It keeps only instruction–response pairs whose sampled answers agree often enough (self-consistency voting).

Using Qwen2.5-72B-Instruct as the generator, the authors built datasets for seven domains, including mathematics, finance, biomedicine and logical reasoning. Smaller open models fine-tuned on this data beat both their base versions and models trained with prior synthesis methods.

This matters because it offers a cheap, scalable way to adapt LLMs to niche domains without collecting any domain data first.

Key points

Fine-tuned models: Llama3-8B, Mistral-7B and Qwen2.5-7B, evaluated on seven domain benchmarks such as GSM8K, Math, CFA, MedQA, PubMedQA, LogiQA and GPQA.
Retrieval setup: The corpus is sampled from five subsets of the Pile (ArXiv, FreeLaw, StackExchange, Wikipedia, Github). It is queried with BM25-style scoring using the task description plus random keyword subsets.
Instruction diversity: Instructions span six Bloom levels: Remember, Understand, Apply, Analyze, Evaluate and Create.
Scaling: Mistral-7B accuracy keeps rising from 1,000 to 6,000 training samples without saturating. On Qwen models of 3B, 7B and 14B parameters, the method beats zero-shot at every size, but the gains shrink as models grow.
Ablations (Mistral-7B, four datasets): Removing keyword expansion or retrieval, the Bloom levels, or self-consistency filtering each lowers accuracy. In one example, accuracy falls from 42.75% to 35.64% without the filtering step.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓Northwestern UniversityEvanston ·

United States41 works · 2,194 citations

Fields, topics and concepts

📚 Shelves 4Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works

🧭 Topics 1Language models & pretraining · 3,320 works

Cite

Cite as Ruiyao Xu, Noelle I. Samia and Han Liu. 2026. DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.176

@inproceedings{xu2026ds2,
  title = {{DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning}},
  author = {Ruiyao Xu, Noelle I. Samia and Han Liu},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.176},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.176}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. arXiv preprint arXiv:2201.07311. — [arXiv:2201.07311](https://arxiv.org/abs/2201.07311) find

Updated 31 October 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-ds2-instruct-domain-specific-data/