research-article

Exploring the Use of Abusive Generative AI Models on Civitai

Authors:

Gareth TysonAuthors Info & Claims

MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

Pages 6949 - 6958

https://doi.org/10.1145/3664647.3681052

Published: 28 October 2024 Publication History

Abstract

The rise of generative AI is transforming the landscape of digital imagery, and exerting a significant influence on online creative communities. This has led to the emergence of AI-Generated Content (AIGC) social platforms, such as Civitai. These distinctive social platforms allow users to build and share their own generative AI models, thereby enhancing the potential for more diverse artistic expression. They also provide artists with the means to showcase their creations (generated from the models), engage in discussions, and obtain feedback, thus nurturing a sense of community. Yet, this openness also raises concerns about the abuse of such platforms, e.g., using models to disseminate deceptive deepfakes or infringe upon copyrights. To explore this, we conduct the first comprehensive empirical study of an AIGC social platform, focusing on its use for generating abusive content. As an exemplar, we construct a comprehensive dataset covering Civitai, the largest available AIGC social platform. Based on this dataset of 87K models and 2M images, we explore the characteristics of content and discuss strategies for moderation to better govern these platforms.

Supplemental Material

MP4 File - Exploring the Use of Abusive Generative AI Models on Civitai

A promotion video for the work "Exploring the Use of Abusive Generative AI Models on Civitai"

Download
30.24 MB

References

[1]

Adem Ait, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2024. HFCommunity: An extraction process and relational database to analyze Hugging Face Hub data. Science of Computer Programming 234 (2024), 103079. https://doi.org/10.1016/j. scico.2024.103079

Digital Library

[2]

Charlotte Bird, Eddie Ungless, and Atoosa Kasirzadeh. 2023. Typology of Risks of Generative Text-to-Image Models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (Montréal, QC, Canada) (AIES '23). Association for Computing Machinery, New York, NY, USA, 396--410. https: //doi.org/10.1145/3600211.3604722

Digital Library

[3]

Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. 2023. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226 (2023).

[4]

Eva Cetinic and James She. 2022. Understanding and Creating Art with AI: Review and Outlook. ACM Trans. Multimedia Comput. Commun. Appl. 18, 2, Article 66 (feb 2022), 22 pages. https://doi.org/10.1145/3475799

Digital Library

[5]

Wei Chen and Shang-Hua Teng. 2017. Interplay between Social Influence and Network Centrality: A Comparative Study on Shapley Centrality and Single-Node-Influence Centrality. In Proceedings of the 26th International Conference on World Wide Web (Perth, Australia) (WWW '17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 967--976. https://doi.org/10.1145/3038912.3052608

Digital Library

[6]

Yiqun T. Chen and James Zou. 2023. TWIGMA: A dataset of AI-Generated Images with Metadata From Twitter. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. https://openreview.net/ forum?id=epUQ40eCzk

[7]

Beomsang Cho, Binh M. Le, Jiwon Kim, Simon Woo, Shahroz Tariq, Alsharif Abuadbba, and Kristen Moore. 2023. Towards Understanding of Deepfake Videos in the Wild. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM '23). Association for Computing Machinery, New York, NY, USA, 4530--4537. https://doi.org/10.1145/ 3583780.3614729

Digital Library

[8]

Civitai. 2023. Civitai's Guide to Buzz! https://education.civitai.com/civitaisguide-to-buzz-%E2%9A%A1/

[9]

Civitai. 2023. How to generate tags like civitai? https://github.com/orgs/civitai/ discussions/506

[10]

Civitai. 2024. Content Rules: Real People. https://civitai.com/content/rules/realpeople

[11]

cloneofsimo. 2023. Low-rank Adaptation for Fast Text-to-Image Diffusion Finetuning. https://github.com/cloneofsimo/lora.

[12]

Ziv Epstein, Aaron Hertzmann, the Investigators of Human Creativity, Memo Akten, Hany Farid, Jessica Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, Robert Mahari, Alex 'Sandy' Pentland, Olga Russakovsky, Hope Schroeder, and Amy Smith. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110--1111. https://doi.org/10.1126/science.adh4451 arXiv:https://www.science.org/doi/pdf/10.1126/science.adh4451

[13]

Yunhe Feng, Pradhyumna Poralla, Swagatika Dash, Kaicheng Li, Vrushabh Desai, and Meikang Qiu. 2023. The Impact of ChatGPT on Streaming Media: A Crowdsourced and Data-Driven Analysis using Twitter and Reddit. In 2023 IEEE 9th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS). 222--227. https://doi.org/10.1109/BigDataSecurity-HPSC-IDS58521.2023.00046

[14]

Giorgio Franceschelli and Mirco Musolesi. 2022. Copyright in generative deep learning. Data & Policy 4 (2022), e17.

[15]

Dilrukshi Gamage, Piyush Ghasiya, Vamshi Bonagiri, Mark E. Whiting, and Kazutoshi Sasahara. 2022. Are Deepfakes Concerning? Analyzing Conversations of Deepfakes on Reddit and Exploring Societal Implications. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 103, 19 pages. https://doi.org/10.1145/3491102.3517446

Digital Library

[16]

Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhan, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2023. CollabCoder: A GPT-Powered Workflow for Collaborative Qualitative Analysis. arXiv preprint arXiv:2304.07366 (2023).

[17]

Robert Gorwa and Michael Veale. 2023. Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediaries. arXiv:2311.12573 [cs.CY]

[18]

Ahmed Hosny, Michael Schwier, Christoph Berger, Evin P Örnek, Mehmet Turan, Phi V Tran, Leon Weninger, Fabian Isensee, Klaus H Maier-Hein, Richard McKinley, Michael T Lu, Udo Hoffmann, Bjoern Menze, Spyridon Bakas, Andriy Fedorov, and Hugo JWL Aerts. 2019. ModelHub.AI: Dissemination Platform for Deep Learning Models. arXiv:1911.13218 [cs.LG]

[19]

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations. https: //openreview.net/forum?id=nZeVKeeFYf9

[20]

Wenxin Jiang, Chingwo Cheung, George K. Thiruvathukal, and James C. Davis. 2023. Exploring Naming Conventions (and Defects) of Pre-trained Deep Learning Models in Hugging Face and Other Model Hubs. arXiv:2310.01642 [cs.SE]

[21]

Abhishek Kumar, Benjamin Finley, Tristan Braud, Sasu Tarkoma, and Pan Hui. 2020. Marketplace for AI Models. arXiv:2003.01593 [cs.CY]

[22]

Max Langenkamp and Daniel N. Yue. 2022. How Open Source Machine Learning Software Shapes AI. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (Oxford, United Kingdom) (AIES '22). Association for Computing Machinery, New York, NY, USA, 385--395. https://doi.org/10.1145/3514094.3534167

Digital Library

[23]

Travis Lloyd, Joseph Reagle, and Mor Naaman. 2023. "There Has To Be a Lot That We're Missing": Moderating AI-Generated Content on Reddit. arXiv preprint arXiv:2311.12702 (2023).

[24]

Zeyu Lu, Di Huang, Lei Bai, Jingjing Qu, Chengyue Wu, Xihui Liu, and Wanli Ouyang. 2023. Seeing is not always believing: Benchmarking Human and Model Perception of AI-Generated Images. arXiv:2304.13023 [cs.AI]

[25]

Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A Holistic Approach to Undesired Content Detection in the Real World. Proceedings of the AAAI Conference on Artificial Intelligence 37, 12 (Jun. 2023), 15009--15018. https://doi.org/10.1609/aaai.v37i12.26752

Digital Library

[26]

Binny Mathew, Anurag Illendula, Punyajoy Saha, Soumya Sarkar, Pawan Goyal, and Animesh Mukherjee. 2020. Hate begets Hate: A Temporal Study of Hate Speech. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 92 (oct 2020), 24 pages. https://doi.org/10.1145/3415163

Digital Library

[27]

Yoshitomo Matsubara. 2023. torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP. arXiv:2310.17644 [cs.CL]

[28]

Jaron Mink, Licheng Luo, Natã M. Barbosa, Olivia Figueira, Yang Wang, and Gang Wang. 2022. Deep Phish: Understanding User Trust Towards Artificially Generated Profiles in Online Social Networks. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 1669--1686. https: //www.usenix.org/conference/usenixsecurity22/presentation/mink

[29]

Yida Mu, Pu Niu, and Nikolaos Aletras. 2022. Identifying and Characterizing Active Citizens who Refute Misinformation in Social Media. In Proceedings of the 14th ACM Web Science Conference 2022 (Barcelona, Spain) (Web-Sci '22). Association for Computing Machinery, New York, NY, USA, 401--410. https://doi.org/10.1145/3501247.3531559

Digital Library

[30]

Mark Newman. 2010. Networks: An Introduction. Oxford University Press. https: //doi.org/10.1093/acprof:oso/9780199206650.001.0001

[31]

OpenAI. 2024. Usage policies. https://openai.com/policies/usage-policies

[32]

Jiameng Pu, Neal Mangaokar, Lauren Kelly, Parantapa Bhattacharya, Kavya Sundaram, Mobin Javed, Bolun Wang, and Bimal Viswanath. 2021. Deepfake Videos in the Wild: Analysis and Detection. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW '21). Association for Computing Machinery, New York, NY, USA, 981--992. https://doi.org/10.1145/3442381.3449978

Digital Library

[33]

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684--10695.

[34]

Similar Web. 2023. Civitai Statistics. https://www.similarweb.com/website/civitai. com.

[35]

Yang Song and Stefano Ermon. 2020. Generative Modeling by Estimating Gradients of the Data Distribution. arXiv:1907.05600 [cs.LG]

[36]

Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. 2023. JourneyDB: A Benchmark for Generative Image Understanding. arXiv:2307.00716 [cs.CV]

[37]

Mina Taraghi, Gianolli Dorcelus, Armstrong Foundjem, Florian Tambon, and Foutse Khomh. 2024. Deep Learning Model Reuse in the Hugging Face Community: Challenges, Benefit and Trends. arXiv:2401.13177 [cs.SE]

[38]

Eddie L Ungless, Björn Ross, and Anne Lauscher. 2023. Stereotypes and Smut: The (Mis) representation of Non-cisgender Identities by Text-to-Image Models. arXiv preprint arXiv:2305.17072 (2023).

[39]

Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2022. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896 (2022).

[40]

Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-Shot Information Extraction via Chatting with ChatGPT. arXiv:2302.10205 [cs.CL]

[41]

Yiluo Wei and Gareth Tyson. 2024. Understanding the Impact of AI Generated Content on Social Media: The Pixiv Case. arXiv:2402.18463 [cs.CY]

[42]

Ziang Xiao, Xingdi Yuan, Q Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. 2023. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. 75--78.

Digital Library

[43]

Danni Xu, Shaojing Fan, and Mohan Kankanhalli. 2023. Combating Misinformation in the Era of Generative AI Models. In Proceedings of the 31st ACM International Conference on Multimedia (MM '23). Association for Computing Machinery, New York, NY, USA, 9291--9298. https://doi.org/10.1145/3581783.3612704

Digital Library

[44]

Digvijay Yadav and Sakina Salmani. 2019. Deepfake: A survey on facial forgery technique using generative adversarial network. In 2019 International conference on intelligent computing and control systems (ICCS). IEEE, 852--857.

[45]

Kai-Cheng Yang, Danishjeet Singh, and Filippo Menczer. 2024. Characteristics and prevalence of fake social media profiles with AI-generated faces. arXiv:2401.02627 [cs.CY]

[46]

SHIH-YING YEH, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, and Yanmin Gong. 2024. Navigating Text-To-Image Customization: From Ly-CORIS Fine-Tuning to Model Evaluation. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=wfzXa8e783

[47]

Yiming Zhu, Ehsan-Ul Haq, Gareth Tyson, Lik-Hang Lee, Yuyang Wang, and Pan Hui. 2023. A Study of Partisan News Sharing in the Russian invasion of Ukraine. arXiv:2311.15294 [cs.SI]

[48]

Yiming Zhu, Zhizhuo Yin, Ehsan-Ul Haq, Lik-Hang Lee, Gareth Tyson, and Pan Hui. 2024. APT-Pipe: An Automatic Prompt-Tuning Tool for Social Computing Data Annotation. arXiv preprint arXiv:2402.01697 (2024).

Cited By

Wei YTyson GCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Understanding the Impact of AI-Generated Content on Social Media: The Pixiv CaseProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680631(6813-6822)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680631
Richards DWorden D(2024)Applications of generative artificial intelligence to influence climate change decisionsnpj Climate Action10.1038/s44168-024-00202-53:1Online publication date: 21-Dec-2024
https://doi.org/10.1038/s44168-024-00202-5

Index Terms

Exploring the Use of Abusive Generative AI Models on Civitai
1. Human-centered computing
  1. Collaborative and social computing
    1. Empirical studies in collaborative and social computing

Recommendations

Understanding the Impact of AI-Generated Content on Social Media: The Pixiv Case
MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

In the last two years, Artificial Intelligence Generated Content (AIGC) has received significant attention, leading to an anecdotal rise in the amount of AIGC being shared via social media platforms. The impact of AIGC and its implications are of key ...
Visions of Destruction: Exploring a Potential of Generative AI in Interactive Art
VINCI '24: Proceedings of the 17th International Symposium on Visual Information Communication and Interaction
This paper explores the potential of generative AI within interactive art, employing a practice-based research approach. It presents the interactive artwork "Visions of Destruction"(2023) by the artist duo Varvara & Mar as a detailed case study, ...
College students social media use and communication network heterogeneity

This study examined whether and how the usage of social media can influence college students' level of network heterogeneity and how network heterogeneity is associated with levels of bridging/bonding social capital and subjective well-being. In ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '24: Proceedings of the 32nd ACM International Conference on Multimedia

October 2024

11719 pages

ISBN:9798400706868

DOI:10.1145/3664647

General Chairs:
Jianfei Cai
Monash University, Australia
,
Mohan Kankanhalli
NUS, Singapore
,
Balakrishnan Prabhakaran
UT Dallas, USA
,
Susanne Boll
University of Oldenburg, Germany
,
Program Chairs:
Ramanathan Subramanian
University of Canberra & IIT Ropar, Australia
,
Liang Zheng
Australian National University, Australia
,
Vivek K. Singh
Rutgers University, USA
,
Pablo Cesar
Centrum Wiskunde & Informatica, Netherlands
,
Lexing Xie
Australian National University, Australia
,
Dong Xu
University of Hong Kong, Hong Kong

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '24

Sponsor:

SIGMM

MM '24: The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne VIC, Australia

Acceptance Rates

MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
94
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)22

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wei YTyson GCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Understanding the Impact of AI-Generated Content on Social Media: The Pixiv CaseProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680631(6813-6822)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680631
Richards DWorden D(2024)Applications of generative artificial intelligence to influence climate change decisionsnpj Climate Action10.1038/s44168-024-00202-53:1Online publication date: 21-Dec-2024
https://doi.org/10.1038/s44168-024-00202-5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten