research-article

Sifter: A Hybrid Workflow for Theme-based Video Curation at Scale

Authors:

Andrés Monroy-Hernández,

Walter S. Lasecki,

Rajan VaishAuthors Info & Claims

IMX '20: Proceedings of the 2020 ACM International Conference on Interactive Media Experiences

Pages 65 - 73

https://doi.org/10.1145/3391614.3393657

Published: 17 June 2020 Publication History

Abstract

User-generated content platforms curate their vast repositories into thematic compilations that facilitate the discovery of high-quality material. Platforms that seek tight editorial control employ people to do this curation, but this process involves time-consuming routine tasks, such as sifting through thousands of videos. We introduce Sifter, a system that improves the curation process by combining automated techniques with a human-powered pipeline that browses, selects, and reaches an agreement on what videos to include in a compilation. We evaluated Sifter by creating 12 compilations from over 34,000 user-generated videos. Sifter was more than three times faster than dedicated curators, and its output was of comparable quality. We reflect on the challenges and opportunities introduced by Sifter to inform the design of content curation systems that need subjective human judgments of videos at scale.

Supplementary Material

p65-chen-supplement (1003-file4.mp4)

Presenting Sifter: Theme-based Video Curation at Scale

Download
68.47 MB

References

[1]

Mark S Ackerman. 2000. The intellectual challenge of CSCW: the gap between social requirements and technical feasibility. Human–Computer Interaction 15, 2-3 (2000), 179–203.

Digital Library

[2]

Georgios Askalidis and Greg Stoddard. 2013. A theoretical analysis of crowdsourced content curation. In The 3rd Workshop on Social Computing and User Generated Content. ACM.

[3]

Werner Bailer, Martin Winter, and Stefanie Wechtitsch. 2017. Learning selection of user generated event videos. In Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing. 1–7.

Digital Library

[4]

Sunil Bandla and Kristen Grauman. 2013. Active learning of an action detector from untrimmed videos. In Computer Vision (ICCV), 2013 IEEE International Conference on. IEEE, 1833–1840.

Digital Library

[5]

Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: from allocative to representational harms in machine learning. Special Interest Group for Computing, Information and Society (SIGCIS) (2017).

[6]

Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. In Conference on Neural Information Processing Systems, Long Beach, CA.

[7]

Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313–322.

Digital Library

[8]

Axel Carlier, Vincent Charvillat, Wei Tsang Ooi, Romulus Grigoras, and Geraldine Morin. 2010. Crowdsourced automatic zoom and scroll for video retargeting. In Proceedings of the 18th ACM international conference on Multimedia. ACM, 201–210.

Digital Library

[9]

Peter McFaul Chapman. 1997. Models of engagement: Intrinsically motivated interaction with multimedia learning software. Ph.D. Dissertation. University of Waterloo.

[10]

Wallace Chipidza. 2016. Negative Behaviors in Online Communities. (2016).

[11]

Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calendar. help: Designing a workflow-based scheduling agent with humans in the loop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2382–2393.

Digital Library

[12]

Edward Curry, Andre Freitas, and Sean O’Riáin. 2010. The Role of Community-Driven Data Curation for Enterprises. Springer US, Boston, MA, 25–47. https://doi.org/10.1007/978-1-4419-7665-9_2

[13]

Peng Dai, Daniel Sabby Weld, 2011. Artificial intelligence for artificial artificial intelligence. In Twenty-Fifth AAAI Conference on Artificial Intelligence.

Digital Library

[14]

Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1013–1022.

Digital Library

[15]

Yanwei Fu, Timothy M Hospedales, Tao Xiang, Jiechao Xiong, Shaogang Gong, Yizhou Wang, and Yuan Yao. 2015. Robust subjective visual property prediction from crowdsourced pairwise labels. IEEE transactions on pattern analysis and machine intelligence 38, 3(2015), 563–577.

[16]

Yanwei Fu, Tao Xiang, Yu-Gang Jiang, Xiangyang Xue, Leonid Sigal, and Shaogang Gong. 2018. Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content. IEEE Signal Processing Magazine 35, 1 (2018), 112–125.

[17]

Philip J Guo, Juho Kim, and Rob Rubin. 2014. How video production affects student engagement: an empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 41–50.

Digital Library

[18]

David Hasler and Sabine E Suesstrunk. 2003. Measuring colorfulness in natural images. In Human vision and electronic imaging VIII, Vol. 5007. International Society for Optics and Photonics, 87–96.

[19]

Yu-Gang Jiang, Yanran Wang, Rui Feng, Xiangyang Xue, Yingbin Zheng, and Hanfang Yang. 2013. Understanding and Predicting Interestingness of Videos. In AAAI.

[20]

Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the session timeout: automatic hierarchical segmentation of search topics in query logs. In Proceedings of the 17th ACM conference on Information and knowledge management. ACM, 699–708.

Digital Library

[21]

Juho Kim, Philip J Guo, Daniel T Seaton, Piotr Mitros, Krzysztof Z Gajos, and Robert C Miller. 2014. Understanding in-video dropouts and interaction peaks inonline lecture videos. In Proceedings of the first ACM conference on Learning@ scale conference. ACM, 31–40.

Digital Library

[22]

Ranjay A Krishna, Kenji Hata, Stephanie Chen, Joshua Kravitz, David A Shamma, Li Fei-Fei, and Michael S Bernstein. 2016. Embracing error to enable rapid crowdsourcing. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 3167–3179.

Digital Library

[23]

Gierad Laput, Walter S Lasecki, Jason Wiese, Robert Xiao, Jeffrey P Bigham, and Chris Harrison. 2015. Zensors: Adaptive, rapidly deployable, human-intelligent sensor feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1935–1944.

Digital Library

[24]

Walter S Lasecki, Mitchell Gordon, Danai Koutra, Malte F Jung, Steven P Dow, and Jeffrey P Bigham. 2014. Glance: Rapidly coding behavioral video with the crowd. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 551–562.

Digital Library

[25]

Walter S Lasecki, Young Chol Song, Henry Kautz, and Jeffrey P Bigham. 2013. Real-time crowd labeling for deployable activity recognition. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1203–1212.

Digital Library

[26]

David Merritt, Jasmine Jones, Mark S Ackerman, and Walter S Lasecki. 2017. Kurator: Using The Crowd to Help Families With Personal Curation Tasks. In CSCW. 1835–1849.

Digital Library

[27]

Vikram Mohanty, David Thames, Sneha Mehta, and Kurt Luther. 2019. Photo sleuth: combining human expertise and face recognition to identify historical portraits. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 547–557.

Digital Library

[28]

Alessandro Ortis, Giovanni Maria Farinella, Valeria D’amico, Luca Addesso, Giovanni Torrisi, and Sebastiano Battiato. 2015. RECfusion: Automatic Video Curation Driven by Visual Content Popularity. In Proceedings of the 23rd ACM International Conference on Multimedia (Brisbane, Australia) (MM ’15). ACM, New York, NY, USA, 1179–1182. https://doi.org/10.1145/2733373.2806311

Digital Library

[29]

Amy Pavel, Dan B Goldman, Björn Hartmann, and Maneesh Agrawala. 2016. VidCrit: Video-based Asynchronous Video Review. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 517–528.

Digital Library

[30]

Reddit. 2019. Reddit Video. https://www.reddit.com/r/videos/.

[31]

Miriam Redi, Neil OHare, Rossano Schifanella, Michele Trevisiol, and Alejandro Jaimes. 2014. 6 seconds of sound and vision: Creativity in micro-videos. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 4272–4279.

Digital Library

[32]

Elliot Salisbury, Sebastian Stein, and Sarvapali Ramchurn. 2015. Crowdar: augmenting live video with a real-time crowd. In Third AAAI Conference on Human Computation and Crowdsourcing.

[33]

David A Shamma, Lyndon Kennedy, Jia Li, Bart Thomee, Haojian Jin, and Jeff Yuan. 2016. Finding weather photos: Community-supervised methods for editorial curation of online sources. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 86–96.

Digital Library

[34]

Snap. 2018. Brand Safety. https://support.snapchat.com/en-US/a/brand-safety.

[35]

Snap. 2018. Discover. https://support.snapchat.com/en-US/a/discover.

[36]

Snap. 2018. Our Story. https://support.snapchat.com/en-US/a/live-story.

[37]

Stanford. 2019. Fair Work. https://fairwork.stanford.edu/.

[38]

Twitter. 2011. Twitter Moments guidelines and principles. https://help.twitter.com/en/rules-and-policies/twitter-moments-guidelines-and-principles.

[39]

Annika Wolff and Paul Mulholland. 2013. Curation, curation, curation. In Proceedings of the 3rd Narrative and Hypertext Workshop. ACM, 1.

Digital Library

[40]

Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. 2018. Every moment counts: Dense detailed labeling of actions in complex videos. International Journal of Computer Vision 126, 2-4 (2018), 375–389.

Digital Library

[41]

Amy X Zhang, Jilin Chen, Wei Chai, Jinjun Xu, Lichan Hong, and Ed Chi. 2018. Evaluation and refinement of clustered search results with the crowd. ACM Transactions on Interactive Intelligent Systems (TiiS) 8, 2(2018), 14.

Cited By

Pei WLikhtenshteyn YYue C(2023)A Tale of Two Communities: Privacy of Third Party App Users in Crowdsourcing - The Case of Receipt TranscriptionProceedings of the ACM on Human-Computer Interaction10.1145/36100447:CSCW2(1-43)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610044

Recommendations

Personal Curation in a Museum

An established body of work in CSCW and related communities studies social and cooperative interaction in museums and cultural heritage sites. A separate and growing body of research in these same communities is developing ways to understand the design ...
Video summarization based on user log enhanced link analysis
MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia

Efficient video data management calls for intelligent video summarization tools that automatically generate concise video summaries for fast skimming and browsing. Traditional video summarization techniques are based on low-level feature analysis, which ...
Personalized video adaptation based on video content analysis
MDM '08: Proceedings of the 9th International Workshop on Multimedia Data Mining: held in conjunction with the ACM SIGKDD 2008

Personalized video adaptation is expected to satisfy individual users' needs on video content. Multimedia data mining plays a significant role of video annotation to meet users' preference on video content. In this paper, a comprehensive solution for ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IMX '20: Proceedings of the 2020 ACM International Conference on Interactive Media Experiences

June 2020

211 pages

ISBN:9781450379762

DOI:10.1145/3391614

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

IMX '20

Sponsor:

IMX '20: ACM International Conference on Interactive Media Experiences

June 17 - 19, 2020

Cornella, Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 69 of 245 submissions, 28%

Upcoming Conference

IMX '25

Sponsor:
sigchi

ACM International Conference on Interactive Media Experiences

June 3 - 6, 2025

Niter?i , Brazil

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
125
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 20 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pei WLikhtenshteyn YYue C(2023)A Tale of Two Communities: Privacy of Third Party App Users in Crowdsourcing - The Case of Receipt TranscriptionProceedings of the ACM on Human-Computer Interaction10.1145/36100447:CSCW2(1-43)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610044

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents