research-article

Have You been Properly Notified? Automatic Compliance Analysis of Privacy Policy Text with GDPR Article 13

Authors:

Meishan ZhangAuthors Info & Claims

WWW '21: Proceedings of the Web Conference 2021

Pages 2154 - 2164

https://doi.org/10.1145/3442381.3450022

Published: 03 June 2021 Publication History

Abstract

With the rapid development of web and mobile applications, as well as their wide adoption in different domains, more and more personal data is provided, consciously or unconsciously, to different application providers. Privacy policy is an important medium for users to understand what personal information has been collected and used. As data privacy protection is becoming a critical social issue, there are laws and regulations being enacted in different countries and regions, and the most representative one is the EU General Data Protection Regulation (GDPR). It is thus important to detect compliance issues among regulations, e.g., GDPR, with privacy policies, and provide intuitive results for data subjects (i.e., users), data collection party (i.e., service providers) and the regulatory authorities. In this work, we target to solve the problem of compliance analysis between GDPR (Article 13) and privacy policies. We format the task into a combination of a sentence classification step and a rule-based analysis step. We manually curate a corpus of 36,610 labeled sentences from 304 privacy policies, and benchmark our corpus with several standard sentence classifiers. We also conduct a rule-based analysis to detect compliance issues and a user study to evaluate the usability of our approach. The web-based tool AutoCompliance is publicly accessible 1.

References

[1]

2011. Data-Driven Documents. https://d3js.org/ (access on 2020.10.17).

[2]

2016. General Data Protection Regulation. https://gdpr-info.eu/ (access on 2020.10.19).

[3]

2018. California Consumer Privacy Act in America. https://oag.ca.gov/privacy/ccpa (access on 2020.10.19).

[4]

2018. Data Protection Act. https://www.gov.uk/data-protection (access on 2020.10.19).

[5]

Waleed Ammar, Shomir Wilson, Norman Sadeh, and Noah A Smith. 2012. Automatic categorization of privacy policies: A pilot study. School of Computer Science, Language Technology Institute, Technical Report CMU-LTI-12-019(2012).

[6]

Vinayshekhar Bannihatti Kumar, Roger Iyengar, Namita Nisal, Yuanyuan Feng, Hana Habib, Peter Story, Sushain Cherivirala, Margaret Hagan, Lorrie Cranor, Shomir Wilson, 2020. Finding a Choice in a Haystack: Automatic Extraction of Opt-Out Statements from Privacy Policy Text. In Proceedings of The Web Conference 2020. 1943–1954.

Digital Library

[7]

Cheng Chang, Huaxin Li, Yichi Zhang, Suguo Du, Hui Cao, and Haojin Zhu. 2019. Automated and Personalized Privacy Policy Extraction Under GDPR Consideration. In WASA 2019. Springer, 43–54.

[8]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning 20, 3 (1995), 273–297.

[9]

Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hosseini, Florian Schaub, and Thorsten Holz. 2018. We Value Your Privacy... Now Take Some Cookies: Measuring the GDPR’s Impact on Web Privacy. arXiv preprint arXiv:1808.05096(2018).

[10]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

[11]

Armin Gerl and Bianca Meier. 2019. The Layered Privacy Language Art. 12–14 GDPR Extension–Privacy Enhancing User Interfaces. Datenschutz und Datensicherheit-DuD 43, 12 (2019), 747–752.

[12]

Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013. Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE workshop on automatic speech recognition and understanding. IEEE, 273–278.

[13]

Nils Gruschka, Vasileios Mavroeidis, Kamer Vishi, and Meiko Jensen. 2018. Privacy issues and data protection in big data: a case study analysis under GDPR. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 5027–5033.

[14]

Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and Understanding the Effectiveness of BERT. In EMNLP/IJCNLP.

[15]

Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. 2018. Polisis: Automated analysis and presentation of privacy policies using deep learning. In USENIX Security 18. 531–548.

[16]

Jasmin Kaur, Rozita A Dara, Charlie Obimbo, Fei Song, and Karen Menard. 2018. A comprehensive keyword analysis of online privacy policies. Information Security Journal 27, 5-6 (2018), 260–275.

[17]

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174.

[18]

Logan Lebanoff and Fei Liu. 2018. Automatic detection of vague words and sentences in privacy policies. In EMNLP 2018.

[19]

Thomas Linden, Rishabh Khandelwal, Hamza Harkous, and Kassem Fawaz. 2018. The privacy policy landscape after the GDPR. arXiv preprint arXiv:1809.08396(2018).

[20]

Fei Liu, Rohan Ramanath, Norman Sadeh, and Noah A Smith. 2014. A step towards usable privacy policy: Automatic alignment of privacy statements. In COLING 2014. 884–894.

[21]

Aleecia M McDonald and Lorrie Faith Cranor. 2008. The cost of reading privacy policies. ISJLP 4(2008), 543.

[22]

Najmeh Mousavi Nejad, Simon Scerri, and Jens Lehmann. 2018. Knight: Mapping privacy policies to gdpr. In European Knowledge Acquisition Workshop. Springer, 258–272.

[23]

Monica Palmirani, Michele Martoni, Arianna Rossi, Cesare Bartolini, and Livio Robaldo. 2018. PrOnto: Privacy ontology for legal reasoning. In International Conference on Electronic Government and the Information Systems Perspective. Springer, 139–152.

Digital Library

[24]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP 2014. 1532–1543.

[25]

Juan Ramos 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, Vol. 242. 133–142.

[26]

Irwin Reyes, Primal Wijesekera, Joel Reardon, Amit Elazari Bar On, Abbas Razaghpanah, Narseo Vallina-Rodriguez, and Serge Egelman. 2018. “Won’t Somebody Think of the Children?” Examining COPPA Compliance at Scale. Proceedings on Privacy Enhancing Technologies 2018, 3(2018), 63–83.

[27]

David Sarne, Jonathan Schler, Alon Singer, Ayelet Sela, and Ittai Bar Siman Tov. 2019. Unsupervised Topic Extraction from Privacy Policies. In WWW 2019. ACM, 563–568.

[28]

Kanthashree Mysore Sathyendra, Florian Schaub, Shomir Wilson, and Norman M Sadeh. 2016. Automatic Extraction of Opt-Out Choices from Privacy Policies. In AAAI Fall Symposia.

[29]

Kanthashree Mysore Sathyendra, Shomir Wilson, Florian Schaub, Sebastian Zimmeck, and Norman Sadeh. 2017. Identifying the provision of choices in privacy policy text. In EMNLP 2017. 2774–2779.

[30]

Peter Story, Sebastian Zimmeck, and Norman Sadeh. 2018. Which apps have privacy policies?. In Annual Privacy Forum. Springer, 3–23.

[31]

Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune BERT for text classification?. In China National Conference on Chinese Computational Linguistics. Springer, 194–206.

Digital Library

[32]

Welderufael B Tesfay, Peter Hofmann, Toru Nakamura, Shinsaku Kiyomoto, and Jetzabel Serna. 2018. I Read but Don’t Agree: Privacy Policy Benchmarking using Machine Learning and the EU GDPR. In Companion Proceedings of the The Web Conference. 163–166.

[33]

Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th annual meeting of the association for computational linguistics: Short papers-volume 2. Association for Computational Linguistics, 90–94.

Digital Library

[34]

Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, 2016. The creation and analysis of a website privacy policy corpus. In ACL 2016 (Volume 1: Long Papers). 1330–1340.

[35]

Jie Yang, Yue Zhang, Linwei Li, and Xingxuan Li. 2017. YEDDA: A lightweight collaborative text span annotation tool. arXiv preprint arXiv:1711.03759(2017).

[36]

Le Yu, Tao Zhang, Xiapu Luo, Lei Xue, and Henry Chang. 2016. Toward automatically generating privacy policy for android apps. IEEE Transactions on Information Forensics and Security 12, 4(2016), 865–880.

Digital Library

[37]

Sebastian Zimmeck, Peter Story, Daniel Smullen, Abhilasha Ravichander, Ziqi Wang, Joel Reidenberg, N Cameron Russell, and Norman Sadeh. 2019. MAPS: Scaling privacy compliance analysis to a million apps. PoPETs 2019, 3 (2019), 66–86.

Cited By

Tocchini MRocha Ide Barros Re Silva JGarcia AZular FMaranhão JSichman J(2025)Classifying Potentially Non-compliant Portuguese Language Sentences Concerning Privacy PoliciesIntelligent Systems10.1007/978-3-031-79038-6_8(107-121)Online publication date: 31-Jan-2025
https://doi.org/10.1007/978-3-031-79038-6_8
Papadimitriou SVirvou MPapadimitriou SVirvou M(2025)General Data Protection Regulation and Adaptive Educational GamesArtificial Intelligence—Based Games as Novel Holistic Educational Environments to Teach 21st Century Skills10.1007/978-3-031-77464-5_9(253-275)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-3-031-77464-5_9
Pan STao ZHoang TZhang DLi TXing ZXu XStaples MRakotoarivelo TLo DBalzarotti DXu W(2024)A NEW HOPEProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699219(5699-5716)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699219
Show More Cited By

Recommendations

The Effect of the GDPR on Privacy Policies: Recent Progress and Future Promise
Special Issue on Analytics for Cybersecurity and Privacy, Part 2 and Regular Papers

The General Data Protection Regulation (GDPR) is considered by some to be the most important change in data privacy regulation in 20 years. Effective May 2018, the European Union GDPR privacy law applies to any organization that collects and processes ...
Democratizing GDPR Compliance: AI-Driven Privacy Policy Interpretation
IC3-2024: Proceedings of the 2024 Sixteenth International Conference on Contemporary Computing

With long privacy policies, users face the challenge of understanding them easily in order to ensure that their privacy rights are upheld. This paper presents an approach for democratizing GDPR compliance through AI-driven privacy policy interpretation. ...
Analyzing GDPR Compliance Through the Lens of Privacy Policy
Heterogeneous Data Management, Polystores, and Analytics for Healthcare
Abstract
With the arrival of the European Union’s General Data Protection Regulation (GDPR), several companies are making significant changes to their systems to achieve compliance. The changes range from modifying privacy policies to redesigning systems ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '21: Proceedings of the Web Conference 2021

April 2021

4054 pages

ISBN:9781450383127

DOI:10.1145/3442381

Editors:
Jure Leskovec
Stanford
,
Marko Grobelnik
Jožef Stefan Institute
,
Marc Najork
Google
,
Jie Tang
Tsinghua University
,
Leila Zia
Wikimedia Foundation

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '21

Sponsor:

SIGWEB

WWW '21: The Web Conference 2021

April 19 - 23, 2021

Ljubljana, Slovenia

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
961
Total Downloads

Downloads (Last 12 months)292
Downloads (Last 6 weeks)11

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tocchini MRocha Ide Barros Re Silva JGarcia AZular FMaranhão JSichman J(2025)Classifying Potentially Non-compliant Portuguese Language Sentences Concerning Privacy PoliciesIntelligent Systems10.1007/978-3-031-79038-6_8(107-121)Online publication date: 31-Jan-2025
https://doi.org/10.1007/978-3-031-79038-6_8
Papadimitriou SVirvou MPapadimitriou SVirvou M(2025)General Data Protection Regulation and Adaptive Educational GamesArtificial Intelligence—Based Games as Novel Holistic Educational Environments to Teach 21st Century Skills10.1007/978-3-031-77464-5_9(253-275)Online publication date: 21-Jan-2025
https://doi.org/10.1007/978-3-031-77464-5_9
Pan STao ZHoang TZhang DLi TXing ZXu XStaples MRakotoarivelo TLo DBalzarotti DXu W(2024)A NEW HOPEProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699219(5699-5716)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699219
Pan SZhang DStaples MXing ZChen JXu XHoang TBalzarotti DXu W(2024)Is it a trap? a large-scale empirical study and comprehensive assessment of online automated privacy policy generators for mobile appsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699218(5681-5698)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699218
Feng YRavichander AYao YZhang SChen RWilson SSadeh NBalzarotti DXu W(2024)Understanding how to inform blind and low-vision users about data privacy through privacy question answering assistantsProceedings of the 33rd USENIX Conference on Security Symposium10.5555/3698900.3699016(2065-2082)Online publication date: 14-Aug-2024
https://dl.acm.org/doi/10.5555/3698900.3699016
Sabur AShowail A(2024)Nudging Data Privacy of Mobile Health Applications in Saudi ArabiaInternational Journal of Information Security and Privacy10.4018/IJISP.34564718:1(1-19)Online publication date: 2-Jul-2024
https://dl.acm.org/doi/10.4018/IJISP.345647
Li SYang ZNan YYu SZhu QYang MLuo BLiao XXu JKirda ELie D(2024)Are We Getting Well-informed? An In-depth Study of Runtime Privacy Notice Practice in Mobile AppsProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670377(1581-1595)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3670377
Smith MTorres-Agüero AGrossman RSen PChen YBorcea CChua TNgo CKa-Wei Lee RKumar RLauw H(2024)A Study of GDPR Compliance under the Transparency and Consent FrameworkProceedings of the ACM Web Conference 202410.1145/3589334.3645618(1227-1236)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645618
Liao SAldeen MYan JCheng LLuo XCai HHu HChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Understanding GDPR Non-Compliance in Privacy Policies of Alexa Skills in European MarketplacesProceedings of the ACM Web Conference 202410.1145/3589334.3645409(1081-1091)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645409
Garza LElluri LPiplai AKotal AGupta DJoshi A(2024)PrivComp-KG: Leveraging KG and LLM for Compliance Verification2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA)10.1109/TPS-ISA62245.2024.00021(97-106)Online publication date: 28-Oct-2024
https://doi.org/10.1109/TPS-ISA62245.2024.00021
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten