Elamur Research⌕
☰
home / papers / 2023 / PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English

PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English

Paper · 2023 · Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian et al.

Summary

Privacy policies are long, legally worded documents that few users read. Earlier NLP work on them treated each task separately, with its own expert-annotated dataset and model. This makes progress hard to compare, and generic pretrained models fit the domain poorly. The authors built PLUE, a unified English benchmark that gathers six existing public tasks into four categories: text classification, question answering, semantic parsing (intent plus slots), and NER. They also crawled a new unlabeled corpus of privacy policies from websites and mobile apps, and used it to continue pretraining BERT, RoBERTa, SpanBERT and Electra, producing PP-* variants. Across every task, the domain-adapted models beat their generic originals. PP-RoBERTa LARGE reached the best average score. LEGAL-BERT, pretrained on general legal text, helped little over plain BERT. Some tasks stay far below human scores, notably APP-350, PrivacyQA and slot filling. PLUE gives the field a shared yardstick, shows that in-domain unlabeled text matters, and points to where models still fail.

Key points

  • Six tasks:
    • OPP-115 and APP-350: multi-label segment classification. For APP-350, the authors added a No_Mention class beside the 30 data-type labels.
    • PrivacyQA: question–sentence relevance treated as binary classification. It includes unanswerable and subjective questions.
    • PolicyQA: extractive span QA.
    • PolicyIE: 4 intents, plus 14 type-I and 4 type-II slots.
    • PI-Extract: NER with COLLECT, NOT_COLLECT, SHARE and NOT_SHARE. These entities can overlap, so collection and sharing are scored separately.
  • Task selection rules: usefulness, diversity of granularity (word, sentence, paragraph), difficulty, each task trainable on one GTX 1080 Ti in at most ten hours, and research-redistributable licenses.
  • Continual pretraining: base models ran 100k steps on V100s, taking 1–2 days (about 9 kg CO₂ reported). Fine-tuning tuned only the learning rate, and results are averaged over 3 seeds.
  • Size of the gains varies by model:
    • SpanBERT improves most: APP-350 F1 rises from 32.8 to 61.9, and OPP-115 from 62.8 to 78.1.
    • The MLM-based BERT and RoBERTa gain moderately.
    • PP-RoBERTa LARGE reaches 74.5 F1 on APP-350.
  • Remaining gap: on PrivacyQA, the best F1 is about 41, versus 68.9 for humans.

Interesting ideas

  • I1. The benchmark's entry criteria include a hard compute budget and a license check, so anyone can rerun it and share it legally.
  • I2. LEGAL-BERT served as a near-domain baseline and showed that text from an adjacent domain is not a substitute for the target domain.
  • I3. Continual pretraining helped the weakest starting models most, so the benefit of adaptation depends on the original training objective.
  • I4. PI-Extract encodes negation in the label itself (NOT_COLLECT versus COLLECT), and because labels overlap it scores the families separately.
  • I5. PrivacyQA recasts QA as judging whether each policy sentence is relevant, and it keeps unanswerable questions in the data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Our evaluation suites could adopt the same admission rule: every test set must train or run within a fixed budget and carry a redistributable license.
  • I2 (learn): When adapting a component to a domain, also train on a neighbouring domain as a control, to check that gains come from the target text itself.
  • I3 (tree): Domain silver data could be tested on both the perceptron parser and gparse, to see which parser actually needs it rather than assuming equal benefit.
  • I4 (frames): Frames should carry explicit polarity, so that "we do not share X" yields a negated share frame rather than a missing one, with collection and sharing evaluated as separate families.
  • I5 (answer): Readers could first mark which sentences of the text world support an answer, and be scored on correctly abstaining when none do.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations🎓University of VirginiaCharlottesville · United States36 works · 9,730 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian and Kai-Wei Chang. 2023. PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-short.31

@inproceedings{chi2023plue,
  title = {{PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English}},
  author = {Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian and Kai-Wei Chang},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.31},
  url = {https://doi.org/10.18653/v1/2023.acl-short.31}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wasi Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, and Kai-Wei Chang. 2021. Intent classification and slot filling for privacy policies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4402–4417, Online. Association for Computational Linguistics. ↗
  2. Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics. ↗
  3. Ryan Amos, Gunes Acar, Eli Lucherini, Mihir Kshirsagar, Arvind Narayanan, and Jonathan Mayer. 2021. Privacy policies over time: Curation and analysis of a million-document dataset. In Proceedings of the Web Conference 2021, WWW ’21, page 2165–2176, New York, NY, USA. Association for Computing Machinery. Calculated using https://mlco2.github.io/impact, based on a total of 100 hours of training on Tesla V100 and Amazon Web Services as the provider. ↗
  4. Markus Bayer, Philipp Kuehn, Ramin Shanehsaz, and Christian Reuter. 2022. Cysecbert: A domain-adapted language model for the cybersecurity domain. arXiv preprint arXiv:2212.02974. ↗
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. find
  6. Jaspreet Bhatia and Travis D Breaux. 2015. Towards an information type lexicon for privacy policies. In 2015 IEEE eighth international workshop on requirements engineering and law (RELAW), pages 19–24. IEEE. find
  7. Jaspreet Bhatia, Morgan C Evans, Sudarshan Wadkar, and Travis D Breaux. 2016. Automated extraction of regulated information types using hyponymy relations. In 2016 IEEE 24th International Requirements Engineering Conference Workshops (REW), pages 19–25. IEEE. find
  8. Duc Bui, Kang G. Shin, Jong-Min Choi, and Junbum Shin. 2021. Automated extraction and presentation of data practices in privacy policies. Proceedings on Privacy Enhancing Technologies, 2021(2):88–110. ↗
  9. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904, Online. Association for Computational Linguistics. ↗
  10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. find
  11. Federal Trade Commission et al. 2012. Protecting consumer privacy in an era of rapid change. FTC report. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  13. Joshua Gluck, Florian Schaub, Amy Friedman, Hana Habib, Norman Sadeh, Lorrie Faith Cranor, and Yuvraj Agarwal. 2016. How short is too short? implications of length and framing on the effectiveness of privacy notices. In Twelfth Symposium on Usable Privacy and Security ( { SOUPS } 2016), pages 321–340. find
  14. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics. ↗
  15. Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. 2018. Poli-sis: Automated analysis and presentation of privacy policies using deep learning. In 27th { USENIX } Security Symposium ( { USENIX } Security 18), pages 531–548. find
  16. Mitra Bokaei Hosseini, Sudarshan Wadkar, Travis D Breaux, and Jianwei Niu. 2016. Lexical similarity of information type hypernyms, meronyms and synonyms in privacy policies. In 2016 AAAI Fall Symposium Series. find
  17. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  18. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. ↗
  19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  20. Fei Liu, Rohan Ramanath, Norman Sadeh, and Noah A. Smith. 2014. A step towards usable privacy policy: Automatic alignment of privacy statements. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 884–894, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  22. Florencia Marotta-Wurgler. 2015. Does “notice and choice” disclosure regulation work? an empirical study of privacy policies,”. In Michigan Law: Law and Economics Workshop. find
  23. Kanthashree Mysore Sathyendra, Shomir Wilson, Florian Schaub, Sebastian Zimmeck, and Norman Sadeh. 2017. Identifying the provision of choices in privacy policy text. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2774–2779, Copenhagen, Denmark. Association for Computational Linguistics. find
  24. Md Rizwan Parvez, Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2022. Retrieval enhanced data augmentation for question answering on privacy policies. arXiv preprint arXiv:2204.08952. ↗
  25. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32. ↗
  26. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  27. Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. ELLE: Efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2789–2810, Dublin, Ireland. Association for Computational Linguistics. find
  28. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  29. Rohan Ramanath, Fei Liu, Norman Sadeh, and Noah A. Smith. 2014. Unsupervised alignment of privacy policies using hidden Markov models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 605–610, Baltimore, Maryland. Association for Computational Linguistics. find
  30. Priyanka Ranade, Aritran Piplai, Anupam Joshi, and Tim Finin. 2021. Cybert: Contextualized embeddings for the cybersecurity domain. In 2021 IEEE International Conference on Big Data (Big Data), pages 3334–3342. IEEE. find
  31. Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. 2021. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):1–13. ↗
  32. Abhilasha Ravichander, Alan W Black, Thomas Norton, Shomir Wilson, and Norman Sadeh. 2021. Breaking down walls of text: How can NLP benefit consumer privacy? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4125–4140, Online. Association for Computational Linguistics. find
  33. Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019. Question answering for privacy policies: Combining computational and legal perspectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4947–4958, Hong Kong, China. Association for Computational Linguistics. ↗
  34. Norman Sadeh, Alessandro Acquisti, Travis D Breaux, Lorrie Faith Cranor, Aleecia M McDonald, Joel R Reidenberg, Noah A Smith, Fei Liu, N Cameron Russell, Florian Schaub, et al. 2013. The usable privacy policy project. Technical report, Technical Report, CMU-ISR-13-119. find
  35. Kanthashree Mysore Sathyendra, Florian Schaub, Shomir Wilson, and Norman Sadeh. 2016. Automatic extraction of opt-out choices from privacy policies. In 2016 AAAI Fall Symposium Series. find
  36. Yan Shvartzshanider, Ananth Balashankar, Thomas Wies, and Lakshminarayanan Subramanian. 2018. RECIPE: Applying open domain question answering to privacy policies. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 71–77, Melbourne, Australia. Association for Computational Linguistics. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. ↗
  39. Henry William. 2020. Do web apps and mobile apps need separate privacy policies? find
  40. Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N. Cameron Russell, Thomas B. Norton, Eduard Hovy, Joel Reidenberg, and Norman Sadeh. 2016a. The creation and analysis of a website privacy policy corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330–1340, Berlin, Germany. Association for Computational Linguistics.
  41. Shomir Wilson, Florian Schaub, Rohan Ramanath, Norman Sadeh, Fei Liu, Noah A Smith, and Frederick Liu. 2016b. Crowdsourcing annotations for websites’ privacy policies: Can it really work? In Proceedings of the 25th International Conference on World Wide Web, pages 133–143. International World Wide Web Conferences Steering Committee. find
  42. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. ↗
  43. Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel Reidenberg, N Cameron Russell, and Norman Sadeh. 2019. Maps: Scaling privacy compliance analysis to a million apps. Proceedings on Privacy Enhancing Technologies, 2019(3):66–86. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chi-2023-plue-language-understanding-evaluation-benchmark/