📄 Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Risks

Summary

Copyright Detective is an interactive forensic tool for collecting evidence that LLMs may reproduce copyrighted material. Three problems make such auditing hard. Random decoding makes results unstable. Safety alignment can hide memorized content. Model updates may suppress recall without truly removing it. The authors therefore treat copyright assessment as an evidence-gathering process, not a single yes/no classification.

The system combines several modules in one extensible framework:

• Content recall: verbatim reproduction from text snippets or whole uploaded documents.
• Persuasive jailbreak: rhetorical rewrites of blocked requests, checked so they keep the original intent.
• Knowledge memorization: open-ended questions about a book's content.
• Unlearning verification: comparison of model versions.

Outputs are scored against user-supplied ground truth with ROUGE and other similarity metrics. Repeated sampling (inference scaling) catches leaks that appear only occasionally.

In experiments with GPT-4o-mini, the "Pathos" persuasion strategy raised ROUGE-L leakage from roughly 0.1 to roughly 0.7. The tool is useful for rights holders, AI red teams and educators. The authors stress that its scores are technical indicators, not legal rulings.

Key points

• The system is described as having five forensic modules and works mainly with black-box access. Where token probabilities are available (e.g., GPT-Neo), it adds Min-K% Prob; where weights are open (e.g., Llama), it supports white-box representational analysis.
• Similarity metrics include ROUGE (ROUGE-L in particular), Jaccard Similarity, Levenshtein Distance and semantic similarity, plus counts of matched, missed and extra tokens.
• On GPT-4o-mini, the Pathos strategy moved the ROUGE-L leakage distribution from about 0.1 to about 0.7, turning a consistent refusal into a probabilistic leakage risk.
• For unlearning, PCA on hidden states was applied to tofu_ft_llama2-7b versus Llama-2-7b-hf, using 10 TOFU queries. The strongest representation drift appeared in layers 29–31, but the authors note this shows change, not proven erasure.
• The tool is deployed as a Streamlit web app. Case studies cover Harry Potter, Pride and Prejudice, The Hobbit and The Great Gatsby. For now the tool concentrates on works of literature and does not yet handle source code or content in other modalities.