M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection
Paper · 2023 · Yuxia Wang, Jonibek Mansurov, Petar Ivanov et al.
Summary
People can barely tell LLM output from human writing, and earlier detection resources were narrow: mostly English-only, built around one generator such as ChatGPT, or limited to a single domain like news. This paper presents M4, a benchmark for black-box detection, where a detector sees only the text and has no access to the model that may have produced it. The authors combined existing human-written corpora (Wikipedia, WikiHow, Reddit ELI5, arXiv, PeerRead, Baike/Web QA, RuATD, news) with outputs from six LLMs, each prompted in several styles. They applied light cleaning and checked quality at the model level. Seven detectors were then evaluated across domains, generators, languages and time periods. The main finding is that detectors transfer poorly to domains or generators not seen during training, and in those cases they tend to label machine text as human-written. The problem is therefore far from solved. M4 is publicly released and was later extended into SemEval-2024 Task 8, giving the field a shared testbed for building more robust detectors.
Key points
- Size: About 147k human–machine parallel items: roughly 102k in English and 45k in other languages (9k each for Chinese, Russian and Bulgarian; 6k each for Urdu, Indonesian and Arabic). Over 10M additional non-parallel human texts are included.
- Generators: GPT-4, ChatGPT, text-davinci-003, Cohere, Dolly-v2 and BLOOMz 176B. Each was given 2–8 prompts in different styles, such as simple explanations or a formal academic tone.
- Length and cleaning: Texts have a minimum length of 1,000 characters, or the equivalent for non-English languages. Formatting artifacts such as bullet points, repeated newlines and Wikipedia references were removed so detectors cannot rely on trivial cues.
- Generalization: Cross-domain and cross-generator settings remain difficult. BLOOMz output was especially hard to separate from human text.
- Availability: Data and code are at github.com/mbzuai-nlp/M4. The authors plan to keep adding generators, domains and languages. Licensing follows the original corpora and LLM terms.
Institutions
United Kingdom256 works · 123,373 citations🎓Technische Universität DarmstadtDarmstadt · 
Germany114 works · 36,326 citations🎓New York University Abu DhabiAbu Dhabi · 
United Arab Emirates124 works · 4,709 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citationsFields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works
Cite
Cite as Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych and Preslav Nakov. 2023. M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection. arXiv:2305.14902.
@misc{wang2023m4,
title = {{M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection}},
author = {Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych and Preslav Nakov},
year = {2023},
doi = {10.48550/arxiv.2305.14902},
eprint = {2305.14902},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2305.14902}
}References
- Bias: Biases may exist in both human-written and machine-generated texts, and it is possible that our M4 dataset may be influenced by biases from human collection, thus affecting the detection outcomes. We leave the analysis of such biases to our future work. find
- Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. find
- Feasibility of Black-Box Machine-Generated Text Detection find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-m4-multi-generator-multi-domain/