Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3613904.3642868acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

ReelFramer: Human-AI Co-Creation for News-to-Video Translation

Published: 11 May 2024 Publication History

Abstract

Short videos on social media are the dominant way young people consume content. News outlets aim to reach audiences through news reels—short videos conveying news—but struggle to translate traditional journalistic formats into short, entertaining videos. To translate news into social media reels, we support journalists in reframing the narrative. In literature, narrative framing is a high-level structure that shapes the overall presentation of a story. We identified three narrative framings for reels that adapt social media norms but preserve news value, each with a different balance of information and entertainment. We introduce ReelFramer, a human-AI co-creative system that helps journalists translate print articles into scripts and storyboards. ReelFramer supports exploring multiple narrative framings to find one appropriate to the story. AI suggests foundational narrative details, including characters, plot, setting, and key information. ReelFramer also supports visual framing; AI suggests character and visual detail designs before generating a full storyboard. Our studies show that narrative framing introduces the necessary diversity to translate various articles into reels, and establishing foundational details helps generate scripts that are more relevant and coherent. We also discuss the benefits of using narrative framing and foundational details in content retargeting.

Supplemental Material

MP4 File - Video Presentation
Video Presentation
Transcript for: Video Presentation

References

[1]
Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O Riedl. 2021. Automated storytelling via causal, commonsense plot ordering. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 5859–5867.
[2]
Craig Batty, Stayci Taylor, Louise Sawtell, and Bridget Conor. 2017. Script development: Defining the field. Journal of Screenwriting 8, 3 (2017), 225–247.
[3]
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 4762–4779.
[4]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 1877–1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
[5]
Judy Carter. 2001. The Comedy Bible: From Stand-up to Sitcom–The Comedy Writer’s Ultimate "How To" Guide. Touchstone, New York, New York, USA. 368 pages.
[6]
David Caswell and Konstantin Dörr. 2018. Automated Journalism 2.0: Event-driven narratives: From simple descriptions to real stories. Journalism practice 12, 4 (2018), 477–496.
[7]
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2023. Art or artifice? large language models and the false promise of creativity. arXiv preprint arXiv:2309.14556 (2023).
[8]
Shizhe Chen, Bei Liu, Jianlong Fu, Ruihua Song, Qin Jin, Pingping Lin, Xiaoyu Qi, Chunting Wang, and Jin Zhou. 2019. Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences. In Proceedings of the 27th ACM International Conference on Multimedia (Nice, France) (MM ’19). Association for Computing Machinery, New York, NY, USA, 2236–2244. https://doi.org/10.1145/3343031.3350571
[9]
Jaemin Cho, Abhay Zala, and Mohit Bansal. 2023. Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3043–3054.
[10]
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 209, 19 pages. https://doi.org/10.1145/3491102.3501819
[11]
Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly Short Attention Spans in Neural Language Modeling. In International Conference on Learning Representations. https://openreview.net/forum?id=ByIAPUcee
[12]
Lloyd S Davis, Bienvenido León, Michael J Bourk, Lei Zhu, and Wiebke Finkler. 2022. Infotainment May Increase Engagement with Science but It Can Decrease Perceptions of Seriousness. Sustainability 14, 17 (2022), 10659.
[13]
Greg Dean. 2000. Step by Step to Stand-Up Comedy. Heinemann Drama, Portsmouth, New Hampshire. 208 pages.
[14]
Angel Fiallos, Carlos Fiallos, and Stalin Figueroa. 2021. Tiktok and education: Discovering knowledge through learning videos. In 2021 Eighth International Conference on EDemocracy & EGovernment (ICEDEG). IEEE, 172–176.
[15]
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing Using Language Models. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1002–1019. https://doi.org/10.1145/3532106.3533533
[16]
Joy Paul Guilford. 1967. The nature of human intelligence. (1967).
[17]
Matthew Guzdial and Mark Riedl. 2019. An interaction framework for studying co-creative ai. arXiv preprint arXiv:1903.09709 (2019).
[18]
Stephen Halliwell. 1998. Aristotle’s poetics. University of Chicago Press.
[19]
Tony Harcup and Deirdre O’neill. 2017. What is news? News values revisited (again). Journalism studies 18, 12 (2017), 1470–1488.
[20]
Clare Hayes, Katherine Stott, Katie J Lamb, and Glenn A Hurst. 2020. “Making every second count”: Utilizing TikTok and systems thinking to facilitate scientific public engagement and contextualization of chemistry at home.
[21]
Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Storeoboard: Sketching Stereoscopic Storyboards. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4587–4598. https://doi.org/10.1145/2858036.2858079
[22]
Wynford Hicks, Adams Sally, Harriett Gilbert, Tim Holmes, and Jane Bentley. 2016. Writing for journalists. Routledge.
[23]
Sally Holloway. 2010. The Serious Guide to Joke Writing: How To Say Something Funny About Anything. Bookshaker, Great Yarmouth, UK. 207 pages.
[24]
David Holz. 2022. Midjourney. https://midjourney.com/
[25]
Automated Insights. 2016. The Associated Press Leaps Forward. http://go.automatedinsights.com/the-associated-press-leaps-forward.html.
[26]
David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11.
[27]
Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. PromptMaker: Prompt-Based Prototyping with Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 35, 8 pages. https://doi.org/10.1145/3491101.3503564
[28]
Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proceedings of the seventh international conference on computational creativity. 77–84.
[29]
Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in Large Language Models. In Proceedings of The ACM Collective Intelligence Conference. 12–24.
[30]
Boyang Li and Mark Riedl. 2010. An offline planning approach to game plotline adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 6. 45–50.
[31]
Hang Li. 2022. Language Models: Past, Present, and Future. Commun. ACM 65, 7 (jun 2022), 56–63. https://doi.org/10.1145/3490443
[32]
Vivian Liu, Han Qiao, and Lydia Chilton. 2022. Opal: Multimodal Image Generation for News Illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 73, 17 pages. https://doi.org/10.1145/3526113.3545621
[33]
Tao Long, Dorothy Zhang, Grace Li, Batool Taraif, Samia Menon, Kynnedy Simone Smith, Sitong Wang, Katy Ilonka Gero, and Lydia B. Chilton. 2023. Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Media. In Proceedings of the 14th Conference on Computational Creativity(ICCC ’23). Association for Computational Creativity.
[34]
Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J. Cai. 2020. Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376739
[35]
Adyasha Maharana, Darryl Hannan, and Mohit Bansal. 2022. Storydall-e: Adapting pretrained text-to-image transformers for story continuation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXVII. Springer, 70–87.
[36]
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. 2023. Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 355, 34 pages. https://doi.org/10.1145/3544548.3581225
[37]
Enrico Motta, Enrico Daga, Andreas L Opdahl, and Bjørnar Tessem. 2020. Analysis and design of computational news angles. IEEE Access 8 (2020), 120613–120626.
[38]
Santiago Negrete Yankelevich and Nora Angelica Morales Zaragoza. 2014. The apprentice framework: planning and assessing creativity. Fifth International Conference on Computational Creativity.
[39]
Nic Newman. 2022. How publishers are learning to create and distribute news on TikTok. (2022).
[40]
Nic Newman, Richard Fletcher, Robertson Craig T, Kirsten Eddy, and Rasmus Kleis Nielsen. 2022. Digital News Report 2022. https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2022 Last accessed 30 Mar 2023.
[41]
OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]
[42]
Savvas Petridis, Nicholas Diakopoulos, Kevin Crowston, Mark Hansen, Stan Jastrzebski, Keren Henderson, Jeffrey V. Nickerson, and Lydia B. Chilton. 2023. AngleKindling: Supporting Journalistic Angle Ideation with Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
[43]
Joe Pompeo. 2023. “Meet people where they are”: Legacy media wants a piece of that gen Z tiktok mojo. https://www.vanityfair.com/news/2023/02/tiktok-media-new-york-times-washington-post
[44]
Horst Pottker. 2003. News and its communicative quality: the inverted pyramid—when and why did it appear?Journalism Studies 4, 4 (2003), 501–511.
[45]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 (2022).
[46]
Victor Raskin. 2009. The Primer of Humor Research. De Gruyter, Berlin, Germany. 673 pages.
[47]
Laurel Richardson. 1990. Writing strategies: Reaching diverse audiences. Vol. 21. Sage Publications.
[48]
Mark O. Riedl and R. Michael Young. 2010. Narrative Planning: Balancing Plot and Character. J. Artif. Int. Res. 39, 1 (sep 2010), 217–268.
[49]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2021. High-Resolution Image Synthesis with Latent Diffusion Models. arxiv:2112.10752 [cs.CV]
[50]
Reijo Savolainen. 2022. Infotainment as a hybrid of information and entertainment: a conceptual analysis. Journal of documentation 78, 4 (2022), 953–970.
[51]
R.C. Schank and R.P. Abelson. 1977. Scripts, Plans, Goals, and Understanding: An Inquiry Into Human Knowledge Structures. L. Erlbaum Associates. https://books.google.com/books?id=YZ99AAAAMAAJ
[52]
Johannes CP Schmid and Johannes CP Schmid. 2021. Narrative Framing: Storytelling, Structures, and Perspectives. Frames and Framing in Documentary Comics (2021), 201–274.
[53]
Hanieh Shakeri, Carman Neustaedter, and Steve DiPaola. 2021. SAGA: Collaborative Storytelling with GPT-3. In Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (Virtual Event, USA) (CSCW ’21). Association for Computing Machinery, New York, NY, USA, 163–166. https://doi.org/10.1145/3462204.3481771
[54]
Yang Shi, Nan Cao, Xiaojuan Ma, Siji Chen, and Pei Liu. 2020. EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376520
[55]
Sangho Suh, Jian Zhao, and Edith Law. 2022. CodeToon: Story Ideation, Auto Comic Generation, and Structure Mapping for Code-Driven Storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST ’22). Association for Computing Machinery, New York, NY, USA, Article 13, 16 pages. https://doi.org/10.1145/3526113.3545617
[56]
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems 27 (2014).
[57]
Jane Thier. 2022. How a 28-year-old is transforming the ‘male, pale, and stale’ personal finance industry to become everyone’s ‘Rich BFF’. https://fortune.com/2022/11/27/how-vivian-tu-rich-bff-tiktok-became-successful-personal-finance/ Last accessed 30 Mar 2023.
[58]
Sebastián Valenzuela, Martina Piña, and Josefina Ramírez. 2017. Behavioral effects of framing on social media users: How conflict, economic, human interest, and morality frames drive news sharing. Journal of communication 67, 5 (2017), 803–826.
[59]
Patti M Valkenburg, Holli A Semetko, and Claes H De Vreese. 1999. The effects of news frames on readers’ thoughts and recall. Communication research 26, 5 (1999), 550–569.
[60]
Arantxa Vizcaíno-Verdú and Crystal Abidin. 2022. Music challenge memes on TikTok: Understanding in-group storytelling videos. International Journal of Communication 16 (2022), 26.
[61]
Sitong Wang, Savvas Petridis, Taeahn Kwon, Xiaojuan Ma, and Lydia B Chilton. 2023. PopBlends: Strategies for conceptual blending with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–19.
[62]
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems 33 (2020), 5776–5788.
[63]
Yuxin Wang, Jieru Lin, Zhiwei Yu, Wei Hu, and Börje F. Karlsson. 2023. Open-world story generation with structured knowledge enhancement: A comprehensive survey. Neurocomputing 559 (2023), 126792. https://doi.org/10.1016/j.neucom.2023.126792
[64]
Ben S Wasike. 2013. Framing News in 140 Characters: How Social Media Editors Frame the News and Interact with Audiences via Twitter.Global Media Journal: Canadian Edition 6, 1 (2013).
[65]
Daniel S Weld. 1999. Recent advances in AI planning. AI magazine 20, 2 (1999), 93–93.
[66]
Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 359, 10 pages. https://doi.org/10.1145/3491101.3519729
[67]
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 385, 22 pages. https://doi.org/10.1145/3491102.3517582
[68]
Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. The Next Chapter: A Study of Large Language Models in Storytelling. In Proceedings of the 16th International Natural Language Generation Conference. 323–351.
[69]
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 7378–7385.
[70]
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI ’22). Association for Computing Machinery, New York, NY, USA, 841–852. https://doi.org/10.1145/3490099.3511105
[71]
JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
[72]
Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA, Article 5, 30 pages. https://doi.org/10.1145/3586183.3606800
[73]
Andrew Zhu, Lara Martin, Andrew Head, and Chris Callison-Burch. 2023. CALYPSO: LLMs as Dungeon Master’s Assistants. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 19. 380–390.

Cited By

View all
  • (2024)Exploring the Impact of AI-Generated Images on Political News Perception and UnderstandingCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681907(565-571)Online publication date: 11-Nov-2024
  • (2024)PodReels: Human-AI Co-Creation of Video Podcast TeasersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661591(958-974)Online publication date: 1-Jul-2024
  • (2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. creativity support tools
  2. generative AI
  3. narratives
  4. scriptwriting
  5. short videos
  6. storyboarding

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

CHI '24

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25
CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1,268
  • Downloads (Last 6 weeks)169
Reflects downloads up to 24 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Exploring the Impact of AI-Generated Images on Political News Perception and UnderstandingCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681907(565-571)Online publication date: 11-Nov-2024
  • (2024)PodReels: Human-AI Co-Creation of Video Podcast TeasersProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661591(958-974)Online publication date: 1-Jul-2024
  • (2024)Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI WorkflowProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661587(782-803)Online publication date: 1-Jul-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media