Article

Creating music videos using automatic media analysis

Authors:

Jonathan Foote,

Matthew Cooper,

Andreas GirgensohnAuthors Info & Claims

MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia

Pages 553 - 560

https://doi.org/10.1145/641007.641119

Published: 01 December 2002 Publication History

Abstract

We present methods for automatic and semi-automatic creation of music videos, given an arbitrary audio soundtrack and source video. Significant audio changes are automatically detected; similarly, the source video is automatically segmented and analyzed for suitability based on camera motion and exposure. Video with excessive camera motion or poor contrast is penalized with a high unsuitability score, and is more likely to be discarded in the final edit. High quality video clips are then automatically selected and aligned in time with significant audio changes. Video clips are adjusted to match the audio segments by selecting the most suitable region of the desired length. Besides a fully automated solution, our system can also start with clips manually selected and ordered using a graphical interface. The video is then created by truncating the selected clips (preserving the high quality portions) to produce a video digest that is synchronized with the soundtrack music, thus enhancing the impact of both.

Supplementary Material

JPG File (641119.jpg)

Download
14.56 KB

MPG File (641119.mpg)

Download
28.61 MB

References

[1]

C. Bregler, M. Covell, and M. Slaney. "Video rewrite: Driving visual speech with audio." Computer Graphics Annual Conference Series, 1997

Digital Library

[2]

Boreczky, J. and Rowe, L., "Comparison of Video Shot Boundary Detection Techniques," in Proc. SPIE Conference on Storage and Retrieval for Still Image and Video Databases IV, San Jose, CA, February, 1996, pp. 170--179

[3]

Christel, M., Smith, M., Taylor, C. and Winkler, D., "Evolving Video Skims into Useful Multimedia Abstractions" in CHI 98 Conference Proceedings (Los Angeles, CA), New York: ACM, pp. 171--178, 1998.

Digital Library

[4]

Cooper, M. and Foote, J., "Scene Boundary Detection Via Video Self-Similarity Analysis." Proc. IEEE Intl. Conf. on Image Processing, pp. 378--81, 2001.

[5]

Foote, J., "Automatic Audio Segmentation using a Measure of Audio Novelty." in Proc. of IEEE ICME, vol. I, pp. 452--455, 2000.

[6]

Foote, J., and Uchihashi, S.,"The Beat Spectrum: A New Approach to Rhythm Analysis," submitted to ICME 2001.

[7]

A. Girgensohn, S. Bly, F. Shipman, J. Boreczky, and L. Wilcox. "Home Video Editing Made Easy --- Balancing Automation and User Control." In Human-Computer Interaction INTERACT '01, IOS Press, pp. 464--471, 2001.

[8]

Girgensohn, A., Boreczky, J., Chiu, P., Doherty, J., Foote., J., Golovchinsky, G., Uchihashi, S., and Wilcox, L. (2000), A Semi-Automatic Approach to Home Video Editing, in UIST '00 Proceedings, ACM Press, pp. 81--89.

Digital Library

[9]

Goto, M. and Y. Muraoaka (1994). "A Beat Tracking System for Acoustic Signals of Music," In Proc. ACM Multimedia 1994, San Francisco, ACM.

Digital Library

[10]

Kennedy, H.,"A Floydian Analysis of 'The Wizard of Oz'," in The New York Daily News, May 13, 1997, (also http:// www.straightdope.com/mailbag/mdarkside.htm).

[11]

J. Kruskal and D. Sankoff, "An Anthology of Algorithms and Concepts for Sequence Comparison," in Time Warps, String Edits, and Macromolecules: the Theory and Practice of String Comparison, eds. D. Sankoff and J. Kruskal, CSLI Publications, 1999

[12]

Lienhart, R., "Abstracting Home Video Automatically," in Proc. ACM Multimedia '99 (Part 2), pp. 37--40, 1999.

Digital Library

[13]

Lipscomb, S.D. (1997). "Perceptual measures of visual and auditory cues in film music." JASA101(5, ii), p. 3190 (online version at http://imr.utsa.edu/~lipscomb/JASA97/)

[14]

Lipscomb, S.D. & Kendall R.A. "Perceptual judgment of the relationship between musical and visual components in film." Psychomusicology, 13(1), pp. 60--98, (1994) (online version at http://imr.utsa.edu/~lipscomb/Thesis/thes00.html)

[15]

MPEG Requirements Group. Description of MPEG-7 Content Set, Doc. ISO/MPEG N2467, MPEG Atlantic City Meeting, October 1998.

[16]

muvee AutoProducer, http://www.muvee.com

[17]

W. R. Neuman, "Beyond HDTV: Exploring Subjective Responses to Very High Definition Television"; MIT Media Laboratory Report, July, 1990.

[18]

Pfeiffer, S., Lienhart, R., Fischer, S. and Effelsberg, W., "Abstracting Digital Movies Automatically," in Journal of Visual Communication and Image Representation, 7(4), pp. 345--353, December 1996.

[19]

Sack, W., and Davis, M. "IDIC: Assembling Video Sequences from Story Plans and Content Annotations." In Proc. IEEE International Conference on Multimedia Computing and Systems. Boston, Ma., May 14--19, 1994.

[20]

Scheirer, Eric D. (1998). "Tempo and Beat Analysis of Acoustic Musical Signals." In J. Acoust. Soc. Am.103(1) (Jan 1998), pp. 588--601.

[21]

Smith, M. and Kanade, T., "Video Skimming and Characterization through the Combination of Image and Language Understanding Techniques," in Proc. Computer Vision and Pattern Recognition, pp. 775--781, 1997.

Digital Library

[22]

Suzuki, R. and Iwadate, Y., "Multimedia Montage--Counterpoint synthesis of movies," in Proc. IEEE Multimedia Systems '99, Vol. 1, pp. 433--438, 1999

Digital Library

[23]

Van Trees, H., Detection, Estimation, and Modulation Theory, Pt. 1. J. Wiley and Sons, 1968.

Cited By

Kato JOgata MInoue TGoto MBoll SMu Lee KLuo JZhu WByun HWen Chen CLienhart RMei T(2018)Songle SyncProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3240619(1697-1705)Online publication date: 15-Oct-2018
https://dl.acm.org/doi/10.1145/3240508.3240619
Matsumoto YHarakawa ROgawa THaseyama M(2018)Simultaneous Realization of Multiple Music Video Applications Based on Heterogeneous Network Analysis Via Latent Link Estimation2018 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME.2018.8486474(1-6)Online publication date: Jul-2018
https://doi.org/10.1109/ICME.2018.8486474
Roininen MLeppänen JEronen ACurcio IGabbouj M(2017)Modeling the timing of cuts in automatic editing of concert videosMultimedia Tools and Applications10.1007/s11042-016-3304-776:5(6683-6707)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11042-016-3304-7
Show More Cited By

Recommendations

Automatic music video summarization based on audio-visual-text analysis and alignment
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

In this paper, we propose a novel approach for automatic music video summarization based on audio-visual-text analysis and alignment. The music video is separated into the music and video tracks. For the music track, the chorus is detected based on ...
Automatic music video generation based on temporal pattern analysis
MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia

Music video (MV) is a short film meant to present a visual representation of a popular music song. In this paper, we present a system that automatically generates MV-like videos from personal home videos based on observations that generally there are ...
Automatic summarization of music videos

In this article, we propose a novel approach for automatic music video summarization. The proposed summarization scheme is different from the current methods used for video summarization. The music video is separated into the music track and video ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia

December 2002

683 pages

ISBN:158113620X

DOI:10.1145/641007

Conference Chair:
Lawrence Rowe
UC Berkeley
,
General Chair:
Bernard Merialdo
Institut EURECOM
,
Program Chairs:
Max Muhlhauser
TU Darmstadt
,
Keith Ross
Institut EURECOM
,
Nevenka Dimitrova
Phillips Research

Copyright © 2002 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2002

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MM02

Sponsor:

MM02: ACM Multimedia 2002

December 1 - 6, 2002

Juan-les-Pins, France

Acceptance Rates

MULTIMEDIA '02 Paper Acceptance Rate 46 of 330 submissions, 14%;

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
1,146
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)2

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kato JOgata MInoue TGoto MBoll SMu Lee KLuo JZhu WByun HWen Chen CLienhart RMei T(2018)Songle SyncProceedings of the 26th ACM international conference on Multimedia10.1145/3240508.3240619(1697-1705)Online publication date: 15-Oct-2018
https://dl.acm.org/doi/10.1145/3240508.3240619
Matsumoto YHarakawa ROgawa THaseyama M(2018)Simultaneous Realization of Multiple Music Video Applications Based on Heterogeneous Network Analysis Via Latent Link Estimation2018 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME.2018.8486474(1-6)Online publication date: Jul-2018
https://doi.org/10.1109/ICME.2018.8486474
Roininen MLeppänen JEronen ACurcio IGabbouj M(2017)Modeling the timing of cuts in automatic editing of concert videosMultimedia Tools and Applications10.1007/s11042-016-3304-776:5(6683-6707)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s11042-016-3304-7
Fan JLi WBizzocchi JBizzocchi JPasquier PKuramoto IAndo HWallner G(2016)DJ-MVPProceedings of the 13th International Conference on Advances in Computer Entertainment Technology10.1145/3001773.3001782(1-8)Online publication date: 9-Nov-2016
https://dl.acm.org/doi/10.1145/3001773.3001782
Schindler ARauber A(2016)Harnessing Music-Related Visual Stereotypes for Music Information RetrievalACM Transactions on Intelligent Systems and Technology10.1145/29267198:2(1-21)Online publication date: 25-Oct-2016
https://dl.acm.org/doi/10.1145/2926719
Goto MYoshii KNakano T(2015)Songle Widget: Making Animation and Physical Devices Synchronized with Music Videos on the Web2015 IEEE International Symposium on Multimedia (ISM)10.1109/ISM.2015.64(85-88)Online publication date: Dec-2015
https://doi.org/10.1109/ISM.2015.64
Akiyama YOore SWatters C(2015)Framework for constructing task-space to support novice multimedia authoringMultimedia Tools and Applications10.1007/s11042-014-1911-874:15(6111-6147)Online publication date: 1-Jul-2015
https://dl.acm.org/doi/10.1007/s11042-014-1911-8
Rubin SAgrawala MBenko HDontcheva MWigdor D(2014)Generating emotionally relevant musical scores for audio storiesProceedings of the 27th annual ACM symposium on User interface software and technology10.1145/2642918.2647406(439-448)Online publication date: 5-Oct-2014
https://dl.acm.org/doi/10.1145/2642918.2647406
Cricri FDabov KCurcio IMate SGabbouj M(2014)Multimodal extraction of events and of information about the recording activity in user generated videosMultimedia Tools and Applications10.1007/s11042-012-1085-170:1(119-158)Online publication date: 1-May-2014
https://dl.acm.org/doi/10.1007/s11042-012-1085-1
Outtagarts ASquedin SMartinot O(2014)Cloud-Based Automatic Video Editing Using KeywordsE-Business and Telecommunications10.1007/978-3-662-44791-8_14(228-241)Online publication date: 12-Sep-2014
https://doi.org/10.1007/978-3-662-44791-8_14
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents