research-article

Open access

Automatic Generation of Two-Level Hierarchical Tutorials from Instructional Makeup Videos

Authors:

Maneesh AgrawalaAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 108, Pages 1 - 16

https://doi.org/10.1145/3411764.3445721

Published: 07 May 2021 Publication History

All formats PDF

Abstract

We present a multi-modal approach for automatically generating hierarchical tutorials from instructional makeup videos. Our approach is inspired by prior research in cognitive psychology, which suggests that people mentally segment procedural tasks into event hierarchies, where coarse-grained events focus on objects while fine-grained events focus on actions. In the instructional makeup domain, we find that objects correspond to facial parts while fine-grained steps correspond to actions on those facial parts. Given an input instructional makeup video, we apply a set of heuristics that combine computer vision techniques with transcript text analysis to automatically identify the fine-level action steps and group these steps by facial part to form the coarse-level events. We provide a voice-enabled, mixed-media UI to visualize the resulting hierarchy and allow users to efficiently navigate the tutorial (e.g., skip ahead, return to previous steps) at their own pace. Users can navigate the hierarchy at both the facial-part and action-step levels using click-based interactions and voice commands. We demonstrate the effectiveness of segmentation algorithms and the resulting mixed-media UI on a variety of input makeup videos. A user study shows that users prefer following instructional makeup videos in our mixed-media format to the standard video UI and that they find our format much easier to navigate.

Supplementary Material

Supplementary Materials (3411764.3445721_supplementalmaterials.zip)

Download
57.74 KB

References

[1]

Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan, and Barbara Tversky. 2003. Designing Effective Step-by-step Assembly Instructions. ACM Trans. Graph. 22, 3 (July 2003), 828–837. https://doi.org/10.1145/882262.882352

Digital Library

[2]

Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. 2016. Unsupervised learning from narrated instruction videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, Las Vegas, NV, USA, 4575–4583.

[3]

B. Brown. 2008. Bobbi Brown Makeup Manual: For Everyone from Beginner to Pro. Grand Central Publishing, New York, NY, USA.

[4]

Huiwen Chang, Jingwan Lu, Fisher Yu, and Adam Finkelstein. 2018. PairedCycleGAN: Asymmetric Style Transfer for Applying and Removing Makeup. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, Salt Lake City, UT, USA, 40–48.

[5]

Minsuk Chang, Anh Truong, Oliver Wang, Maneesh Agrawala, and Juho Kim. 2019. How to Design Voice Based Navigation for How-To Videos. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 701, 11 pages. https://doi.org/10.1145/3290605.3300931

Digital Library

[6]

Hung-Jen Chen, Ka-Ming Hui, Szu-Yu Wang, Li-Wu Tsao, Hong-Han Shuai, and Wen-Huang Cheng. 2019. BeautyGlow: On-Demand Makeup Transfer Framework With Reversible Generative Network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, Long Beach, CA, USA, 10042–10050.

[7]

Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: Automatic Generation of Step-by-Step Mixed Media Tutorials. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (Cambridge, Massachusetts, USA) (UIST ’12). Association for Computing Machinery, New York, NY, USA, 93–102. https://doi.org/10.1145/2380116.2380130

Digital Library

[8]

Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. DemoCut: Generating Concise Instructional Videos for Physical Demonstrations. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 141––150. https://doi.org/10.1145/2501988.2502052

Digital Library

[9]

Jonathan D. Denning, William B. Kerr, and Fabio Pellacini. 2011. MeshFlow: Interactive Visualization of Mesh Construction Sequences. In ACM SIGGRAPH 2011 Papers (Vancouver, British Columbia, Canada) (SIGGRAPH ’11). Association for Computing Machinery, New York, NY, USA, Article 66, 8 pages. https://doi.org/10.1145/1964921.1964961

Digital Library

[10]

Logan Fiorella and Richard E Mayer. 2018. What works and doesn’t work with instructional video.

[11]

C. Ailie Fraser, Joy O. Kim, Hijung Valentina Shin, Joel Brandt, and Mira Dontcheva. 2020. Temporal Segmentation of Creative Live Streams. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376437

Digital Library

[12]

Google. 2020. Cloud Natural Language documentation | Cloud Natural Language API. Google Cloud. https://cloud.google.com/natural-language/docs

[13]

Google. 2020. Cloud Speech-to-Text - Speech Recognition | Google Cloud. Google Cloud. https://cloud.google.com/speech-to-text

[14]

Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating Photo Manipulation Tutorials by Demonstration. In ACM SIGGRAPH 2009 Papers (New Orleans, Louisiana) (SIGGRAPH ’09). Association for Computing Machinery, New York, NY, USA, Article 66, 9 pages. https://doi.org/10.1145/1576246.1531372

Digital Library

[15]

Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: Capture, Exploration, and Playback of Document Workflow Histories. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 143–152. https://doi.org/10.1145/1866029.1866054

Digital Library

[16]

Julie Heiser, Doantam Phan, Maneesh Agrawala, Barbara Tversky, and Pat Hanrahan. 2004. Identification and Validation of Cognitive Design Principles for Automated Generation of Assembly Instructions. In Proceedings of the Working Conference on Advanced Visual Interfaces (Gallipoli, Italy) (AVI ’04). ACM, New York, NY, USA, 311–319. https://doi.org/10.1145/989863.989917

Digital Library

[17]

D. Huang, J. J. Lim, L. Fei-Fei, and J. C. Niebles. 2017. Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Computer Vision Foundation / IEEE, Honolulu, HI, USA, 1032–1041.

[18]

De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. 2018. Finding “It”: Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, Salt Lake City, UT, USA, 5948–5957.

[19]

Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, 2017. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE conference on computer vision and pattern recognition. Computer Vision Foundation / IEEE, Honolulu, HI, USA, 7310–7311.

[20]

R. Jones. 2017. Robert Jones’ Makeup Masterclass: A Complete Course in Makeup for All Levels, Beginner to Advanced. Fair Winds Press, Beverly, MA, USA.

[21]

Yoshinari Kameda and Michihiko Minoh. 1996. A human motion estimation method using 3-successive video frames. In International conference on virtual systems and multimedia. VSMM, Gifu, Japan, 135–140.

[22]

Archana Kannan. 2020. Shopping for a beauty product? Try it on with Google. https://blog.google/products/shopping/shopping-beauty-product-try-it-google/

[23]

Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 4017–4026. https://doi.org/10.1145/2556288.2556986

Digital Library

[24]

Bridjet Lee and Kasia Muldner. 2020. Instructional Video Design: Investigating the Impact of Monologue- and Dialogue-Style Presentations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376845

Digital Library

[25]

Rainer W. Lienhart. 1998. Comparison of automatic shot boundary detection algorithms. In Storage and Retrieval for Image and Video Databases VII, Minerva M. Yeung, Boon-Lock Yeo, and Charles A. Bouman (Eds.), Vol. 3656. International Society for Optics and Photonics, SPIE, San Jose, CA, USA, 290 – 301. https://doi.org/10.1117/12.333848

[26]

Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. 2015. What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision. CoRR abs/1503.01558(2015), 1–10. arxiv:1503.01558http://arxiv.org/abs/1503.01558

[27]

Julie Bauer Morrison, Barbara Tversky, and Mireille Betrancourt. 2000. Animation: Does it facilitate learning. In AAAI spring symposium on smart graphics, Vol. 5359. The AAAI Press, Menlo Park, CA, USA, 8.

[28]

Megha Nawhal, Jacqueline B Lang, Greg Mori, and Parmit K Chilana. 2019. VideoWhiz: Non-Linear Interactive Overviews for Recipe Videos. In Graphics Interface. Canadian Information Processing Society, Kingston, Ontario, Canada, 15–1.

[29]

Celie O’Neil-Hart. 2017. Self-directed learning from YouTube - Think with Google. https://www.thinkwithgoogle.com/advertising-channels/video/self-directed-learning-youtube/

[30]

Amy Pavel, Colorado Reed, Björn Hartmann, and Maneesh Agrawala. 2014. Video Digests: A Browsable, Skimmable Format for Informational Lecture Videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology(Honolulu, Hawaii, USA) (UIST ’14). ACM, New York, NY, USA, 573–582. https://doi.org/10.1145/2642918.2647400

Digital Library

[31]

Promise Phan. 2015. ’INSIDE OUT’ Makeup Tutorial (Disgust,Sadness,Joy,Anger & Fear). https://www.youtube.com/watch?v=C1DXqkOCBt0

[32]

Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F. Cohen. 2011. Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology(Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 135–144. https://doi.org/10.1145/2047196.2047213

Digital Library

[33]

Ozan Sener, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. 2015. Unsupervised semantic parsing of video collections. In Proceedings of the IEEE International Conference on Computer Vision. Computer Vision Foundation / IEEE, Santiago, Chile, 4480–4488.

Digital Library

[34]

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision. Computer Vision Foundation / IEEE, Long Beach, CA, USA, 7464–7473.

[35]

Anh Truong, Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2016. QuickCut: An Interactive Tool for Editing Narrated Video. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST ’16). Association for Computing Machinery, New York, NY, USA, 497–507. https://doi.org/10.1145/2984511.2984569

Digital Library

[36]

Sylvaine Tuncer, Barry Brown, and Oskar Lindwall. 2020. On Pause: How Online Instructional Videos Are Used to Achieve Practical Tasks. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376759

Digital Library

[37]

Sylvaine Tuncer, Barry Brown, and Oskar Lindwall. 2020. On Pause: How Online Instructional Videos Are Used to Achieve Practical Tasks. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376759

Digital Library

[38]

Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. 2014. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, Columbus, OH, USA, 1386–1393.

Digital Library

[39]

Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 405–416. https://doi.org/10.1145/2675133.2675219

Digital Library

[40]

Wikipedia contributors. 2020. F1 score. https://en.m.wikipedia.org/wiki/F1_score

[41]

Ann Yuan and Andrey Vakunov. 2020. Face and hand tracking in the browser with MediaPipe and TensorFlow.js. https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html

[42]

Jeffrey M Zacks and Barbara Tversky. 2001. Event structure in perception and conception.Psychological bulletin 127, 1 (2001), 3.

[43]

Jeffrey M Zacks, Barbara Tversky, and Gowri Iyer. 2001. Perceiving, remembering, and communicating structure in events.Journal of experimental psychology: General 130, 1 (2001), 29.

[44]

Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Computer Vision Foundation / IEEE, Long Beach, CA, USA, 3537–3545.

Cited By

Wiethüchter RKalloori SLindlbauer D(2024)Balancing Autonomy: Investigating User-Controlled vs Automated Guidance Systems for Sequential TasksCompanion Proceedings of the 2024 Conference on Interactive Surfaces and Spaces10.1145/3696762.3698056(73-78)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3696762.3698056
Sadprasid BGutwin CBateman S(2024)Improving Video Navigation for Spatial Task Tutorials by Spatially Segmenting and Situating How-To VideosProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682103(1-13)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682103
Nghiem CBousseau ASypesteyn MHoftijzer JAgrawala MTsandilas T(2024)STIVi: Turning Perspective Sketching Videos into Interactive TutorialsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670969(1-13)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670969
Show More Cited By

Index Terms

Automatic Generation of Two-Level Hierarchical Tutorials from Instructional Makeup Videos

Index terms have been assigned to the content through auto-classification.

Recommendations

Automatic makeup based on generative adversarial nets
ICIMCS '18: Proceedings of the 10th International Conference on Internet Multimedia Computing and Service

Automatic makeup aims to compose and synthesize makeup on human face through computer algorithm, which belongs to the field of face image analysis. It plays an important role in interactive entertainment applications, image and video editing and face ...
Anti-makeup: learning a bi-level adversarial network for makeup-invariant face verification
AAAI'18/IAAI'18/EAAI'18: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence

Makeup is widely used to improve facial attractiveness and is well accepted by the public. However, different makeup styles will result in significant facial appearance changes. It remains a challenging problem to match makeup and non-makeup face images. ...
Real-world automatic makeup via identity preservation makeup net
IJCAI'20: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence

This paper focuses on the real-world automatic makeup problem. Given one non-makeup target image and one reference image, the automatic makeup is to generate one face image, which maintains the original identity with the makeup style in the reference ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
1,763
Total Downloads

Downloads (Last 12 months)686
Downloads (Last 6 weeks)75

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wiethüchter RKalloori SLindlbauer D(2024)Balancing Autonomy: Investigating User-Controlled vs Automated Guidance Systems for Sequential TasksCompanion Proceedings of the 2024 Conference on Interactive Surfaces and Spaces10.1145/3696762.3698056(73-78)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3696762.3698056
Sadprasid BGutwin CBateman S(2024)Improving Video Navigation for Spatial Task Tutorials by Spatially Segmenting and Situating How-To VideosProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682103(1-13)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682103
Nghiem CBousseau ASypesteyn MHoftijzer JAgrawala MTsandilas T(2024)STIVi: Turning Perspective Sketching Videos into Interactive TutorialsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670969(1-13)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670969
Wang SNing ZTruong ADontcheva MLi DChilton L(2024)PodReels: Human-AI Co-Creation of Video Podcast TeasersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661591(958-974)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661591
Emerson AEndow STorres C(2024)Anther: Cross-Pollinating Communities of Practice via Video TutorialsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660727(1991-2005)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660727
Yu FZhang PDing XLu TGu N(2024)BNoteHelper: A Note-based Outline Generation Tool for Structured Learning on Video-sharing PlatformsACM Transactions on the Web10.1145/363877518:2(1-30)Online publication date: 12-Mar-2024
https://dl.acm.org/doi/10.1145/3638775
Jin HKim J(2024)CodeTree: A System for Learnersourcing Subgoal Hierarchies in Code ExamplesProceedings of the ACM on Human-Computer Interaction10.1145/36373088:CSCW1(1-37)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637308
Emerson AEndow STorres C(2024)Shared, Shaped, and Stolen: Tracing Sites of Knowledge Transfer across Creative Communities of PracticeProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656199(638-650)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656199
Lin DCaba Heilbron FLee JWang OMartelaro N(2024)Videogenic: Identifying Highlight Moments in Videos with Professional Photographs as a PriorProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656186(328-346)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656186
Yang SVermeulen JFitzmaurice GMatejka J(2024)AQuA: Automated Question-Answering in Software Tutorial Videos with Visual AnchorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642752(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642752
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents