research-article

Analyzing Information Leakage of Updates to Natural Language Models

Authors:

Santiago Zanella-Béguelin,

Lukas Wutschitz,

Olga Ohrimenko,

Marc BrockschmidtAuthors Info & Claims

CCS '20: Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security

Pages 363 - 375

https://doi.org/10.1145/3372297.3417880

Published: 02 November 2020 Publication History

Abstract

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---differential score and differential rank---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

Supplementary Material

MOV File (Copy of CCS2020_fpc318_SantiagoZanella - Brian Hollendyke.mov)

Presentation video

Download
338.94 MB

References

[1]

Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In 23rd ACM SIGSAC Conference on Computer and Communications Security, CCS 2016. ACM, 308--318.

[2]

Galen Andrew, Steve Chien, and Nicolas Papernot. 2020. TensorFlow Privacy. https://github.com/tensorflow/privacy.

[3]

Arm. 2020. TrustZone Technology. https://developer.arm.com/ip-products/security-ip/trustzone

[4]

Raef Bassily, Adam Smith, and Abhradeep Thakurta. 2014. Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014. IEEE Computer Society, 464--473.

[5]

Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine Unlearning. In 42nd IEEE Symposium on Security and Privacy, S&P 2021. IEEE Computer Society. To appear.

[6]

Nicholas Carlini, Chang Liu, Ú lfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Symposium. USENIX Association, 267--284.

[7]

David Cash, Paul Grubbs, Jason Perry, and Thomas Ristenpart. 2015. Leakage-Abuse Attacks Against Searchable Encryption. In 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015. ACM, 668--679.

[8]

Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2020. When Machine Unlearning Jeopardizes Privacy. arxiv: 2005.02205 [cs.CR]

[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Vol. 1. Association for Computational Linguistics, 380--385.

[10]

Cynthia Dwork, Moni Naor, Toniann Pitassi, Guy N. Rothblum, and Sergey Yekhanin. 2010. Pan-Private Streaming Algorithms. In Innovations in Computer Science, ICS 2010. Tsinghua University Press, 66--80.

[11]

Cynthia Dwork and Aaron Roth. 2014. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in Theoretical Computer Science, Vol. 9, 3--4 (2014), 211--407.

Digital Library

[12]

Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. In 22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015. ACM, 1322--1333.

Digital Library

[13]

Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon M. Lin, David Page, and Thomas Ristenpart. 2014. Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing. In 23rd USENIX Security Symposium. USENIX Association, 17--32.

[14]

Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. 2019. Making AI Forget You: Data Deletion in Machine Learning. In Advances in Neural Information Processing Systems 32, NeurIPS 2019. Curran Associates, Inc., 3518--3531.

[15]

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020. IEEE, 9301--9309.

[16]

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten. 2020. Certified Data Removal from Machine Learning Models. In 37th International Conference on Machine Learning, ICML 2020. PMLR. To appear.

[17]

Sepp Hochreiter and Jü rgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, Vol. 9, 8 (1997), 1735--1780.

Digital Library

[18]

Intel. 2020. Software Guard Extensions (SGX). https://software.intel.com/en-us/sgx

[19]

Ken Lang. 1995. NewsWeeder: Learning to Filter Netnews. In 12th International Machine Learning Conference on Machine Learning, ICML 1995. Morgan Kaufmann, 331--339.

[20]

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, Vol. 19, 2 (1993), 313--330.

Digital Library

[21]

H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning Differentially Private Recurrent Language Models. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net.

[22]

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In 5th International Conference on Learning Representations, ICLR 2017. OpenReview.net.

[23]

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. Technical Report. OpenAI.

[24]

Ahmed Salem, Apratim Bhattacharyya, Michael Backes, Mario Fritz, and Yang Zhang. 2019 a. Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning. arxiv: 1904.01067 [cs.CR]

[25]

Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2019 b. ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models. In 26th Annual Network and Distributed System Security Symposium, NDSS 2019. The Internet Society.

[26]

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership Inference Attacks Against Machine Learning Models. In 38th IEEE Symposium on Security and Privacy, S&P 2017. IEEE Computer Society, 3--18.

[27]

Congzheng Song and Vitaly Shmatikov. 2019. Auditing Data Provenance in Text-Generation Models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019. ACM, 196--206.

Digital Library

[28]

S. Song, K. Chaudhuri, and A. D. Sarwate. 2013. Stochastic Gradient Descent with Differentially Private Updates. In 1st IEEE Global Conference on Signal and Information Processing, GlobalSIP 2013. IEEE Computer Society, 245--248.

[29]

European Union. 2016. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation).

[30]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems 30, NIPS 2017. Curran Associates, Inc., 5998--6008.

[31]

Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2018. Diverse Beam Search for Improved Description of Complex Scenes. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018. AAAI Press, 7371--7379.

[32]

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 31st IEEE Computer Security Foundations Symposium, CSF 2018. IEEE Computer Society, 268--282.

[33]

Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent Neural Network Regularization. arxiv: 1409.2329 [cs.NE]

Cited By

Wang XWei Y(2025)Defender: The Possibility of Repairing Jailbreak DefectsProceedings of the 3rd International Conference on Machine Learning, Cloud Computing and Intelligent Mining (MLCCIM2024)10.1007/978-981-96-1694-7_17(198-209)Online publication date: 1-Feb-2025
https://doi.org/10.1007/978-981-96-1694-7_17
Yang HMa ZLiu YLiu XYang BMa J(2025)Updates Leakage Attack Against Private Graph Split LearningAlgorithms and Architectures for Parallel Processing10.1007/978-981-96-1528-5_1(1-21)Online publication date: 15-Feb-2025
https://doi.org/10.1007/978-981-96-1528-5_1
Dong YMu RJin GQi YHu JZhao XMeng JRuan WHuang XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)PositionProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692521(11375-11394)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692521
Show More Cited By

Index Terms

Analyzing Information Leakage of Updates to Natural Language Models
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Natural language generation
  2. Machine learning
2. Security and privacy
  1. Software and application security

Recommendations

What Does it Mean for a Language Model to Preserve Privacy?
FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their ...
Information Leakage from Data Updates in Machine Learning Models
AISec '23: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security

In this paper we consider the setting where machine learning models are retrained on updated datasets in order to incorporate the most up-to-date information or reflect distribution shifts. We investigate whether one can infer information about these ...
Anonymizing sequential releases under arbitrary updates
EDBT '13: Proceedings of the Joint EDBT/ICDT 2013 Workshops

In today's global information society, governments, companies, public and private institutions and even individuals have to cope with growing demands for personal data publication from scientists, statisticians, journalists and many other data ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '20: Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security

October 2020

2180 pages

ISBN:9781450370899

DOI:10.1145/3372297

General Chairs:
Jay Ligatti
University of South Florida, USA
,
Xinming Ou
University of South Florida, USA
,
Program Chairs:
Jonathan Katz
University of Maryland, USA
,
Giovanni Vigna
University of California-Santa Barbara, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CCS '20

Sponsor:

SIGSAC

CCS '20: 2020 ACM SIGSAC Conference on Computer and Communications Security

November 9 - 13, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '25

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 13 - 17, 2025

Taipei , Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
1,243
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)8

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang XWei Y(2025)Defender: The Possibility of Repairing Jailbreak DefectsProceedings of the 3rd International Conference on Machine Learning, Cloud Computing and Intelligent Mining (MLCCIM2024)10.1007/978-981-96-1694-7_17(198-209)Online publication date: 1-Feb-2025
https://doi.org/10.1007/978-981-96-1694-7_17
Yang HMa ZLiu YLiu XYang BMa J(2025)Updates Leakage Attack Against Private Graph Split LearningAlgorithms and Architectures for Parallel Processing10.1007/978-981-96-1528-5_1(1-21)Online publication date: 15-Feb-2025
https://doi.org/10.1007/978-981-96-1528-5_1
Dong YMu RJin GQi YHu JZhao XMeng JRuan WHuang XSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)PositionProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692521(11375-11394)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692521
Dunstan JVakili TMiranda LVillena FAracena CQuiroga TVera PViteri Valenzuela SRocco V(2024)A pseudonymized corpus of occupational health narratives for clinical entity recognition in SpanishBMC Medical Informatics and Decision Making10.1186/s12911-024-02609-w24:1Online publication date: 24-Jul-2024
https://doi.org/10.1186/s12911-024-02609-w
Huang YLi YWu WZhang JLyu M(2024)Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded CredentialsProceedings of the ACM on Software Engineering10.1145/36608181:FSE(2515-2537)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660818
Chen BIvanov NWang GYan QQuek TGao DZhou JCardenas A(2024)Multi-Turn Hidden Backdoor in Large Language Model-powered Chatbot ModelsProceedings of the 19th ACM Asia Conference on Computer and Communications Security10.1145/3634737.3656289(1316-1330)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3634737.3656289
Zeng ZXiang TGuo SHe JZhang QXu GZhang T(2024)Contrast-Then-Approximate: Analyzing Keyword Leakage of Generative Language ModelsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.339253519(5166-5180)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3392535
Wei JFan MJiao WJin WLiu T(2024)BDMMT: Backdoor Sample Detection for Language Models Through Model Mutation TestingIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.337696819(4285-4300)Online publication date: 2024
https://doi.org/10.1109/TIFS.2024.3376968
Jayaraman BGhosh EChase MRoy SDai WEvans D(2024)Combing for Credentials: Active Pattern Extraction from Smart Reply2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00041(1443-1461)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00041
Jain JKaur I(2024)Exploring the Cloud and Fog: Addressing Security Risks in AI Applications2024 2nd International Conference on Advancements and Key Challenges in Green Energy and Computing (AKGEC)10.1109/AKGEC62572.2024.10868487(1-9)Online publication date: 21-Nov-2024
https://doi.org/10.1109/AKGEC62572.2024.10868487
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten