Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3580305.3599340acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article
Free access

Fair Allocation Over Time, with Applications to Content Moderation

Published: 04 August 2023 Publication History

Abstract

In today's digital world, interaction with online platforms is ubiquitous, and thus content moderation is important for protecting users from content that do not comply with pre-established community guidelines. Given the vast volume of content generated online daily, having an efficient content moderation system throughout every stage of planning is particularly important. We study the short-term planning problem of allocating human content reviewers to different harmful content categories. We use tools from fair division and study the application of competitive equilibrium and leximin allocation rules for addressing this problem. On top of the traditional Fisher market setup, we additionally incorporate novel aspects that are of practical importance. The first aspect is the forecasted workload of different content categories, which puts constraints on the allocation chosen by the planner. We show how a formulation that is inspired by the celebrated Eisenberg-Gale program allows us to find an allocation that not only satisfies the forecasted workload, but also fairly allocates the remaining working hours from the content reviewers among all content categories. A fair allocation of oversupply provides a guardrail in cases where the actual workload deviates from the predicted workload. The second practical consideration is time dependent allocation that is motivated by the fact that partners need scheduling guidance for the reviewers across days to achieve efficiency. To address the time component, we introduce new extensions of the various fair allocation approaches for the single-time period setting, and we show that many properties extend in essence, albeit with some modifications. Lastly, related to the time component, we additionally investigate how to satisfy markets' desire for smooth allocation (i.e, an allocation that does not vary much from time to time) so that the switch in staffing is minimized. We demonstrate the performance of our proposed approaches through real-world data obtained from Meta.

Supplementary Material

MP4 File (rtfp0330-2min-promo.mp4)
The promotional video starts by describing the allocation problem faced by Meta for scheduling human reviewers (supply) to different work types related to content moderation tasks (demand). It then presents the challenges and novel aspects of the given supply demand matching problem. - The first novel aspect is over supply, calling for an algorithm that fairly distribute the over supply to different work types. - The second novel aspect is that the demand on each work type is a time series prediction, calling for an allocation as granular as the demand forecast and as smooth as possible. Lastly, the promotional video gives an overview of the solution we proposed for this allocation problem.

References

[1]
Kenneth Arrow and Michael Intriligator. 2005. Handbook of mathematical economics. Technical Report. Elsevier.
[2]
Haris Aziz and Ethan Brown. 2020. Random Assignment Under Bi-Valued Utilities: Analyzing Hylland-Zeckhauser, Nash-Bargaining, and other Rules. arXiv preprint arXiv:2006.15747 (2020).
[3]
Haris Aziz and Simon Rey. 2020. Almost group envy-free allocation of indivisible goods and chores. Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI) (2020).
[4]
Evripidis Bampis, Bruno Escoffier, and Sasa Mladenovic. 2018. Fair resource allocation over time. In AAMAS 2018--17th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 766--773.
[5]
Caitlin Bassett. 2022. Will AI take over content moderation? Mind Matters News (2022). https://mindmatters.ai/2022/01/will-ai-take-over-content-moderation/
[6]
Ioannis Caragiannis, David Kurokawa, Hervé Moulin, Ariel D Procaccia, Nisarg Shah, and Junxing Wang. 2019. The unreasonable fairness of maximum Nash welfare. ACM Transactions on Economics and Computation (TEAC), Vol. 7, 3 (2019), 1--32.
[7]
Edmund Eisenberg and David Gale. 1959. Consensus of subjective probabilities: The pari-mutuel method. The Annals of Mathematical Statistics, Vol. 30, 1 (1959), 165--168.
[8]
Facebook. 2022. Facebook Community Standards. https://transparency.fb.com/policies/community-standards/ Online; accessed 7-February-2022.
[9]
Duncan Karl Foley. 1966. Resource allocation and the public sector. Yale University.
[10]
Rupert Freeman, Sujoy Sikdar, Rohit Vaish, and Lirong Xia. 2019. Equitable allocations of indivisible goods. arXiv preprint arXiv:1905.10656 (2019).
[11]
Evrard Garcelon, Vashist Avadhanula, Alessandro Lazaric, et al. 2021. Top K Ranking for Multi-Armed Bandit with Noisy Evaluations. arXiv preprint arXiv:2112.06517 (2021).
[12]
Tarleton Gillespie. 2020. Content moderation, AI, and the question of scale. Big Data & Society, Vol. 7, 2 (2020), 2053951720943234.
[13]
Daniel Haimovich, Dima Karamshuk, Thomas J Leeper, Evgeniy Riabenko, and Milan Vojnovic. 2020. Scalable Prediction of Information Cascades over Arbitrary Time Horizons. arXiv preprint arXiv:2009.02092 (2020).
[14]
Daniel Halpern, Ariel D Procaccia, Alexandros Psomas, and Nisarg Shah. 2020. Fair division with binary valuations: One rule to rule them all. In International Conference on Web and Internet Economics. Springer, 370--383.
[15]
Kamal Jain and Vijay V Vazirani. 2010. Eisenberg--Gale markets: algorithms and game-theoretic properties. Games and Economic Behavior, Vol. 70, 1 (2010), 84--106.
[16]
Shagun Jhaver, Iris Birman, Eric Gilbert, and Amy Bruckman. 2019. Human-machine collaboration for content regulation: The case of Reddit Automoderator. ACM Transactions on Computer-Human Interaction (TOCHI), Vol. 26, 5 (2019), 1--35.
[17]
David Kurokawa, Ariel D Procaccia, and Nisarg Shah. 2018. Leximin allocations in the real world. ACM Transactions on Economics and Computation (TEAC), Vol. 6, 3--4 (2018), 1--24.
[18]
Rahul Makhijani, Parikshit Shah, Vashist Avadhanula, Caner Gocmen, Nicolás E Stier-Moses, and Julián Mestre. 2021. QUEST: Queue Simulation for Content Moderation at Scale. arXiv preprint arXiv:2103.16816 (2021).
[19]
Meta. 2022. Community Standards Enforcement Report. https://transparency.fb.com/data/community-standards-enforcement/ Online; accessed 31-January-2023.
[20]
John Nash. 1953. Two-Person Cooperative Games. Econometrica, Vol. 21, 1 (1953), 128--140.
[21]
Viet-An Nguyen, Peibei Shi, Jagdish Ramakrishnan, Udi Weinsberg, Henry C Lin, Steve Metz, Neil Chandra, Jane Jing, and Dimitris Kalimeris. 2020. CLARA: confidence of labels and raters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2542--2552.
[22]
Justin Osofsky. 2019. Our Commitment to Our Content Reviewers. (2019). https://about.fb.com/news/2019/02/commitment-to-content-reviewers/
[23]
Ioannis Panageas, Thorben Tröbst, and Vijay V Vazirani. 2021. Combinatorial Algorithms for Matching Markets via Nash Bargaining: One-Sided, Two-Sided and Non-Bipartite. arXiv preprint arXiv:2106.02024 (2021).
[24]
Benjamin Plaut and Tim Roughgarden. 2020. Almost envy-freeness with general valuations. SIAM Journal on Discrete Mathematics, Vol. 34, 2 (2020), 1039--1068.
[25]
Guy Rosen. 2022. Community Standards Enforcement Report, Third Quarter 2022. (2022). https://about.fb.com/news/2022/11/integrity-and-transparency-reports-q3-2022/
[26]
Katie Schoolov. 2021. Why content moderation costs billions and is so tricky for Facebook, Twitter, YouTube and others. CNBC (2021). https://www.cnbc.com/2021/02/27/content-moderation-on-social-media.html
[27]
Hal R Varian et al. 1974. Equity, envy, and efficiency. Journal of Economic Theory, Vol. 9, 1 (1974), 63--91.
[28]
Stephen J Wilson. 1998. Fair division using linear programming. preprint, Departement of Mathematics, Iowa State University (1998).

Cited By

View all
  • (2024)Verifying Proportionality in Temporal VotingProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663122(2246-2248)Online publication date: 6-May-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 2023
5996 pages
ISBN:9798400701030
DOI:10.1145/3580305
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. content moderation
  2. fair allocation
  3. market equilibrium

Qualifiers

  • Research-article

Conference

KDD '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)723
  • Downloads (Last 6 weeks)24
Reflects downloads up to 10 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Verifying Proportionality in Temporal VotingProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3663122(2246-2248)Online publication date: 6-May-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media