📄 Detecting Sensitive Personal Information in Japanese Pre-Training Corpora for Large Language Models

Summary

The paper tackles a gap in Japanese LLM data curation: web-crawled pre-training corpora may contain sensitive personal data that Japan's APPI classifies as special care-required personal information (SCPI), yet no Japanese work had studied detecting it, and legal restrictions mean no public SCPI dataset exists. The authors build one from a Japanese Common Crawl subset (llm-jp-corpus-v3). They first keep texts that mention person names, using an NER model. Small open LLMs (Swallow 8B, Gemma 9B) then screen these candidates, and only texts both models agree on go to GPT-OSS (120B) for final labeling. Annotators fix outputs the large model gets wrong. Using this data, they train fast classifiers, including traditional models such as SVC and LGBM. These models identify SCPI-related category information well and cost far less than LLM- or Transformer-based classification, so they suit a first filtering pass over huge corpora. SHAP analysis shows that many influential features are irrelevant noise. A further difficulty is linking the identifying information to the sensitive content, which often sit far apart in a text. The dataset stays private for legal and ethical reasons.

Key points

• There are 12 labels in total: a non-SCPI class plus the 11 SCPI categories defined by APPI and its Cabinet Order, namely creed, race, disability, social status, medical history, diagnostic information, medical treatment, criminal record, criminal proceedings, crime victimization, and juvenile protection. For analysis purposes only, an additional LGBT category was introduced.
• The source corpus was 2.61M texts sampled from llm-jp-corpus-v3/ja/ja_cc/level0, averaging 2,474 characters. NER found person names in 1.06M of them.
• Combining the small models raised screening precision. Swallow alone kept 9,220 texts at 0.32 precision, while agreement between Swallow and Gemma (0.8 confidence threshold) kept 2,373 texts at 0.55.
• All LLM annotation ran locally under APPI's academic-research exception. Neither the data nor the classifier is publicly released, partly over misuse concerns.
• Limitations:
  • Only name-based identification is covered.
  • Race, creed, and social status were not evaluated.
  • The classifier cannot learn any SCPI that the annotating LLMs fail to detect.