Elamur Research⌕
☰
home / papers / 2016 / The Creation and Analysis of a Website Privacy Policy Corpus

The Creation and Analysis of a Website Privacy Policy Corpus

Summary

Website privacy policies are legally binding, yet most users neither read nor understand them, and regulators cannot review them at scale. Automating the extraction of key details could help, but suitable annotated data has been scarce. The authors address this gap with the OPP-115 Corpus: 115 English-language privacy policies from US websites, about 267K words in total. The policies were sampled across 15 DMOZ sectors and annotated in fine detail by trained law students using a custom web tool. Domain experts iteratively developed an annotation scheme with ten data-practice categories. Each practice is described by category-specific attributes, and attributes can be linked to supporting text spans. The paper also tests automatic category labeling of paragraph-level segments. An SVM performs best (micro-averaged F1 0.66), ahead of an HMM and logistic regression. The corpus is the first large-scale, fine-grained annotation of privacy policies. It suggests that at least part of the annotation work can be automated, which opens paths toward summarization and tools that make policies easier for users to understand.

Key points

  • Corpus size: OPP-115 contains 115 policies (266,713 words) with 23,194 data practices, 128,347 attributes and 102,576 annotated text spans. Each document was labeled by 3 annotators, drawn from a pool of 10.

  • Policy sampling:

    • Top Google Trends queries from May 2015 yielded 1,799 unique websites.
    • These were grouped into 15 DMOZ top-level sectors. "World" was dropped and "Regional" was restricted to the US.
    • Eight sites were chosen per sector, two at random from each popularity-rank quartile.
    • Each site was checked for an English policy and a US company, and rejected sites were replaced by re-draws from the same quartile.

  • Annotation scheme: There are ten categories:

    • First Party Collection/Use
    • Third Party Sharing/Collection
    • User Choice/Control
    • User Access, Edit & Deletion
    • Data Retention
    • Data Security
    • Policy Change
    • Do Not Track
    • International & Specific Audiences
    • Other

    Each category has mandatory and optional attributes, and every attribute allows the value "Unspecified."

  • Classification results: Micro-averaged F1 for segment category labeling was LR 0.58, HMM 0.60 and SVM 0.66. SVM reached F1 0.75 on First Party Collection/Use, the most frequent category, and 1.0 on Do Not Track.

  • Availability: The dataset can be downloaded from usableprivacy.org/data.

Interesting ideas

  • I1. Stratified popularity sampling: Selecting documents by sector and by rank quartile within each sector gives a corpus that covers both large, legally polished sources and small, niche ones. This approach transfers to any domain-specific corpus where quality and style vary with popularity.
  • I2. An explicit "Unspecified" value: Allowing annotators to record that the text says nothing about an attribute turns vagueness into usable data, instead of forcing guesses. This matters for any text that is often deliberately non-committal, such as legal, policy or government documents.
  • I3. Structured records grounded in evidence spans: Each annotation is a structured record with category-specific attributes, and each attribute can be tied to a text span. The result supports both classification and extraction tasks and keeps every label traceable to its source text.
  • I4. Experts design, trained non-experts annotate: Domain experts built the scheme through repeated rounds on real documents, and skilled annotators applied it. This division balances label quality against cost, and the authors suggest shifting further toward crowdsourcing once coarse labels can be predicted automatically.
  • I5. Hybrid human–machine pipeline: Automatic coarse labeling could route segments to humans for fine-grained work in a single category, with confidence scores flagging uncertain predictions. Clustering policies by their data practices could also reveal sector outliers, such as a bank that collects health information. Both ideas could be adapted to auditing other kinds of document collections.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Law

 Institutions

🎓Stanford UniversityStanford · United States248 works · 214,804 citations🎓Carnegie Mellon UniversityPittsburgh · United States278 works · 138,810 citations🎓Columbia UniversityNew York · United States118 works · 44,744 citations🎓Fordham UniversityNew York · United States2 works · 574 citations

Cite

Cite as Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N. Cameron Russell, Thomas B. Norton, Eduard Hovy, Joel Reidenberg and Norman Sadeh. 2016. The Creation and Analysis of a Website Privacy Policy Corpus. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1126

@inproceedings{wilson2016creation,
  title = {{The Creation and Analysis of a Website Privacy Policy Corpus}},
  author = {Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N. Cameron Russell, Thomas B. Norton, Eduard Hovy, Joel Reidenberg and Norman Sadeh},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1126},
  url = {https://doi.org/10.18653/v1/p16-1126}
}

References

43 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ngo Xuan Bach, Nguyen Le Minh, Tran Thi Oanh, and Akira Shimazu. 2013. A two-phase framework for learning logical structures of paragraphs in legal articles. ACM Transactions on Asian Language Infor-Trevor Bench-Capon, Michał Araszkiewicz, Kevin Ashley, Katie Atkinson, Floris Bex, Filipe Borges, Daniele Bourcier, Paul Bourgine, Jack G Conrad, Enrico Francesconi, et al. 2012. A History of AI and Law in 50 Papers: 25 Years of the International Conference on AI and Law. Artificial Intelligence and Law, 20(3):215–319. find
  2. Guido Boella, Luigi Di Caro, Michele Graziadei, Loredana Cupi, Carlo Emilio Salaroglio, Llio Humphreys, Hristo Konstantinov, Kornel Marko, Livio Robaldo, Claudio Ruffini, et al. 2015. Linking legal open data: Breaking the accessibility and language barrier in European legislation and case law. In Proceedings of the 15th International Conference on Artificial Intelligence and Law, ICAIL ’15, pages 171–175. ACM. find
  3. Kalina Bontcheva, Hamish Cunningham, Ian Roberts, Angus Roberts, Valentin Tablan, Niraj Aswani, and Genevieve Gorrell. 2013. GATE teamware: a web-based, collaborative text annotation framework. Language Resources and Evaluation, 47(4):1007– 1029. find
  4. Travis D. Breaux and Florian Schaub. 2014. Scaling requirements extraction to the crowd. In RE’14: Proceedings of the 22nd IEEE International Requirements Engineering Conference, RE’ 14, pages 163–172, Washington, DC, USA, August. IEEE Society Press. find
  5. Travis D. Breaux, Hanan Hibshi, and Ashwini Rao. 2013. Eddy, a formal language for specifying and analyzing data flow specifications for conflicting privacy requirements. Requirements Engineering, 19(3):281–307, September. find
  6. Jean Carletta. 1996. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254, June. our card
  7. F.H. Cate. 2010. The limits of notice and choice. IEEE Security Privacy, 8(2):59–62, March. find
  8. Parvathi Chundi and Pranav M. Subramaniam. 2014. An approach to analyze web privacy policy documents. In KDD Workshop on Data Mining for Social Good. find
  9. Elisa Costante, Yuanhao Sun, Milan Petković, and Jerry den Hartog. 2012. A machine learning solution to assess privacy policy completeness. In Proceedings of the 2012 ACM Workshop on Privacy in the Electronic Society, WPES ’12, pages 91–96. ACM. find
  10. Elisa Costante, Jerry den Hartog, and Milan Petković. 2013. What websites know about you: Privacy policy analysis using information extraction. In Roberto Di Pietro, Javier Herranz, Ernesto Damiani, and Radu State, editors, Data Privacy Management and Autonomous Spontaneous Security, volume 7731 of Lecture Notes in Computer Science, pages 146–159. Springer. find
  11. Lorrie Faith Cranor, Kelly Idouchi, Pedro Giovanni Leon, Manya Sleeper, and Blase Ur. 2013. Are they actually any different? Comparing thousands of financial institutions’ privacy practices. In Workshop on the Economics of Information Security, WEIS ’13, June. find
  12. Michael Curtotti and Eric McCreath. 2013. A right to access implies a right to know: An open online platform for research on the readability of law. J. Open Access L., 1:1–56. find
  13. Tatiana Ermakova, Benjamin Fabian, and Eleonora Babina. 2015. Readability of privacy policies of healthcare websites. In 12. find
  14. Internationale Tagung Wirtschaftsinformatik (Wirtschaftsinformatik 2015). find
  15. Federal Trade Commission. 2012. Protecting consumer privacy in an era of rapid change: Recommendations for businesses and policymakers. http://www.ftc.gov/reports. Last accessed: June 22, 2016. ↗
  16. Enrico Francesconi, Simonetta Montemagni, Wim Peters, and Daniela Tiscornia. 2010. Semantic processing of legal texts: Where the language of law meets the law of language, volume 6036 of Lecture Notes in Artificial Intelligence. Springer. find
  17. Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Combining different summarization techniques for legal text. In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, HYBRID ’12, pages 115– 123. ACL. find
  18. Google. 2015. Google Trends. https://www.google.com/trends/. Last accessed: June 22, 2016. ↗
  19. Candice Hoke, Lorrie Cranor, Pedro Leon, and Alyssa Au. 2015. Are they worth reading? An in-depth analysis of online tracker’s privacy policies. I/S: A Journal of Law and Policy for the Information Society, 11(2):325–404, April. find
  20. Carlos Jensen and Colin Potts. 2004. Privacy policies as decision-making tools: An evaluation of online privacy notices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’04, pages 471–478. ACM. find
  21. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053. ↗ our card
  22. Fei Liu, Rohan Ramanath, Norman Sadeh, and Noah A. Smith. 2014. A step towards usable privacy policy: Automatic alignment of privacy statements. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, COLING ’14, pages 884–894. ANLP, August. find
  23. Lars Mahler. 2015. What is NLP and why should lawyers care? http://www.lawpracticetoday.org/article/ nlp-lawyers/, February. Last accessed: June 22, 2016. ↗
  24. Aaron K. Massey, Jacob Eisenstein, Annie I. Antón, and Peter P. Swire. 2013. Automated text mining for requirements analysis of policy documents. In 21st IEEE International Requirements Engineering Conference, RE ’13, pages 4–13. IEEE. find
  25. Aleecia M. McDonald and Lorrie Faith Cranor. 2008. The cost of reading privacy policies. I/S: J Law & Policy Info. Soc., 4(3):540–561. find
  26. Gabriele Meiselwitz. 2013. Readability assessment of policies and procedures of social networking sites. In A. Ant Ozok and Panayiotis Zaphiris, editors, 5th International conference, OCSC 2013, Held as Part of HCI International 2013, Lecture Notes in Computer Science, pages 67–75. Springer, January. find
  27. Vytautas Mickevicius, Tomas Krilavicius, and Vaidas Morkevicius. 2015. Classification of short legal Lithuanian texts. In The 5th Workshop on Balto-Slavic Natural Language Processing, BSNLP ’15, pages 106–111. ACM. find
  28. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. find
  29. President’s Concil of Advisors on Science and Technology. 2014. Big data and privacy: a technological perspective. Report to the President, Executive Office of the President, May. find
  30. Rohan Ramanath, Fei Liu, Norman Sadeh, and Noah A. Smith. 2014. Unsupervised alignment of privacy policies using hidden markov models. In Proceedings of the Annual Meeting of the Association of Computational Linguistics, ACL ’14, pages 605–610. ACL, June. find
  31. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45– 50. ELRA, May. find
  32. Joel R. Reidenberg, Travis Breaux, Lorrie Faith Cranor, Brian French, Amanda Grannis, James T. Graves, Fei Liu, Aleecia McDonald, Thomas B. Norton, Rohan Ramanath, N. Cameron Russell, Norman Sadeh, and Florian Schaub. 2015a. Disagreeable privacy policies: Mismatches between meaning and users’ understanding. Berkeley Technology Law Journal, 30(1):39–88. find
  33. Joel R. Reidenberg, N. Cameron Russell, Alexander J. Callen, Sophia Qasir, and Thomas B. Norton. 2015b. Privacy harms and the effectiveness of the notice and choice framework. I/S: J Law & Policy Info. Soc., 11(2). find
  34. Anthony J. Viera and Joanne M. Garrett. 2005. Understanding interobserver agreement: the kappa statistic. Family Medicine, 37(5):360–363, 5. find
  35. Rigo Wenning, Matthias Schunter, Lorrie Cranor, B. Dobbs, S. Egelman, G. Hogben, J. Humphrey, M. Langheinrich, M. Marchiori, M. Presler-Marshall, J. Reagle, and D. A. Stampley. 2006. The platform for privacy preferences 1.1 (P3P 1.1) specification. find
  36. Shomir Wilson, Florian Schaub, Rohan Ramanath, Norman Sadeh, Fei Liu, Noah A. Smith, and Frederick Liu. 2016. Crowdsourcing annotations for websites’ privacy policies: Can it really work? In Proceedings of the 25th World Wide Web Conference, WWW ’13, pages 133–143. find
  37. Sebastian Zimmeck and Steven M. Bellovin. 2014. Privee: An architecture for automatically analyzing web privacy policies. In 23rd USENIX Security Symposium, USENIX Security ’14, pages 1–16. USENIX Association, August. find
  38. Joel R. Reidenberg, Jaspreet Bhatia, Travis Breaux, and Thomas B. Norton. 2016. Automated comparisons of ambiguity in privacy policies and the impact of regulation. Social Science Research Network Working Paper Series, January. find
  39. Norman Sadeh, Alessandro Acquisti, Travis D. Breaux, Lorrie Faith Cranor, Aleecia M. Mcdonald, Joel R. Reidenberg, Noah A. Smith, Fei Liu, N. Cameron Russell, Florian Schaub, and Shomir Wilson. 2013. The usable privacy policy project: Combining crowdsourcing, machine learning and natural language processing to semi-automatically answer those privacy questions users care about. Technical Report CMU-ISR-13-119, Carnegie Mellon University. find
  40. Giovanni Sartor and Antonino Rotolo. 2013. AI and law. In Agreement Technologies, pages 199–207. Springer. find
  41. Jaromı́r Šavelka and Kevin D Ashley. 2015. Transfer of predictive models for classification of statutory texts in multi-jurisdictional settings. In Proceedings of the 15th International Conference on Artificial Intelligence and Law, ICAIL ’15, pages 216–220. ACM. find
  42. Florian Schaub, Rebecca Balebako, Adam L. Durity, and Lorrie Faith Cranor. 2015. A design space for effective privacy notices. In Eleventh Symposium On Usable Privacy and Security, SOUPS ’15, pages 1–17. USENIX Association, July. find
  43. John W. Stamey and Ryan A. Rossi. 2009. Automatically identifying relations in privacy policies. In 27th ACM International Conference on Design of Communication, SIGDOC ’09, pages 233–238. ACM. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wilson-2016-creation-analysis-website-privacy-policy/