tutorial

ReEnact: Sketch based Choreographic Design from Archival Dance Footage

Authors:

Manuel J. Fonseca,

John CollomosseAuthors Info & Claims

ICMR '14: Proceedings of International Conference on Multimedia Retrieval

Pages 313 - 320

https://doi.org/10.1145/2578726.2578766

Published: 01 April 2014 Publication History

Abstract

We describe a novel system for synthesising video choreography using sketched visual storyboards comprising human poses (stick men) and action labels. First, we describe an algorithm for searching archival dance footage using sketched pose. We match using an implicit representation of pose parsed from a mix of challenging low and high fidelity footage. In a training pre-process we learn a mapping between a set of exemplar sketches and corresponding pose representations parsed from the video, which are generalized at query-time to enable retrieval over previously unseen frames, and over additional unseen videos. Second, we describe how a storyboard of sketched poses, interspersed with labels indicating connecting actions, may be used to drive the synthesis of novel video choreography from the archival footage.

We demonstrate both our retrieval and synthesis algorithms over both low fidelity PAL footage from the UK Digital Dance Archives (DDA) repository of contemporary dance, circa 1970, and over higher-definition studio captured footage.

References

[1]

M. Andriluka, S. Roth, and B. Schiele. Pictoral structures revisited: People detection and articulated pose estimation. In Proc. Computer Vision and Pattern Recognition, 2009.

[2]

Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary amp; region segmentation of objects in n-d images. In Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, volume 1, pages 105 --112 vol.1, 2001.

[3]

C. Bregler and M. Covell. Video rewrite: Driving visual speech with audio. SIGGRAPH, pages 353--360, 1997.

Digital Library

[4]

T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy optical flow estimation based on a theory for warping. pages 25--36. Springer, 2004.

[5]

Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang. Mindfinder: interactive sketch-based image search on millions of images. In ACM Multimedia, pages 1605--1608, 2010.

Digital Library

[6]

J. Collomosse, G. Mcneill, and Y. Qian. Storyboard sketches for content based video retrieval. In ICCV, 2009.

[7]

N. Dalal and B. Triggs. Histograms of Oriented Gradients for Human Detection. CVPR 2005, 1:886--893, 2005.

Digital Library

[8]

A. del Bimbo and P. Pala. Visual image retrieval by elastic matching of user sketches. 19(2):121--132, February 1997.

Digital Library

[9]

M. Eichner and V. Ferrari. Better appearance models for pictorial structures. In Proc. British Machine Vision Conf. (BMVC), 2009.

[10]

M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. A descriptor for large scale image retrieval based on sketched feature lines. In SBIM, pages 29--36, 2009.

Digital Library

[11]

M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa. Sketch-based image retrieval: Benchmark and bag-of-features descriptors. In IEEE TVCG, volume 99, 2010.

[12]

T. Ezzat and G. Geiger. Trainable videorealistic speech animation. ACM Transactions on Graphics (TOG), 21(3):388--398, July 2002.

Digital Library

[13]

V. Ferrari, M. J. Marín-Jiménez, and A. Zisserman. Pose search: Retrieving people using their pose. In CVPR, 2009.

[14]

M. J. Fonseca, S. James, and J. P. Collomosse. Skeletons from sketches of dancing poses. In VL/HCC, pages 247--248, 2012.

[15]

J. Harel, C. Koch, and P. Perona. Graph-based visual saliency. In Advances in Neural Information Processing Systems 19, pages 545--552. MIT Press, 2007.

[16]

L. J. Heyer, S. Kruglyak, and S. Yooseph. Exploring expression data: identification and analysis of coexpressed genes. Genome Res., 9:1106--15, 1999.

[17]

R. Hu and J. Collomosse. Motion-sketch based video retrieval using a trellis levenshtein distance. In Intl. Conf. on Pattern Recognition (ICPR), August 2010.

Digital Library

[18]

R. Hu, S. James, and J. Collomosse. Annotated free-hand sketches for video retrieval using object semantics and motion. In Proc. Multimedia Models, 2012.

Digital Library

[19]

A. J. Hunt and A. W. Black. Unit selection in a concatenative speech synthesis system. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'96), 1:373--376, 1996.

Digital Library

[20]

C. E. Jacobs, A. Finkelstein, and D. H. Salesin. Fast multi-resolution image querying. In Proc. ACM SIGGRAPH, pages 277--286, Aug. 1995.

Digital Library

[21]

N. Jammalamadaka, A. Zisserman, M. Eichner, V. Ferrari, and C. V. Jawahar. Video retrieval by mimicking poses. In ICMR, page 34, 2012.

Digital Library

[22]

L. Kovar, M. Gleicher, and F. Pighin. Motion graphs. ACM Transactions on Graphics, 21(3):473--482, July 2002.

Digital Library

[23]

Y. J. Lee, C. L. Zitnick, and M. F. Cohen. Shadowdraw: real-time user guidance for freehand drawing. In ACM SIGGRAPH 2011 papers, SIGGRAPH '11, pages 27:1--27:10, 2011.

Digital Library

[24]

C. Liu, D. Wang, X. Liu, C. Wang, L. Zhang, and B. Zhang. Robust semantic sketch based specific image retrieval. In Proc. Intl. Conf. and Multimedia Expo, 2010.

[25]

F. Mokhtarian and A. K. Mackworth. A theory of multiscale, curvature-based shape representation for planar curves. IEEE Trans. Pattern Anal. Mach. Intell., 14:789--805, August 1992.

Digital Library

[26]

P. Pérez, M. Gangnet, and A. Blake. Poisson image editing. In ACM SIGGRAPH 2003 Papers, SIGGRAPH '03, pages 313--318, New York, NY, USA, 2003. ACM.

Digital Library

[27]

R. Ren and J. Collomosse. Visual sentences for pose retrieval over low-resolution cross-media dance collections. IEEE Transactions on Multimedia, Accepted 2012.

[28]

C. Rother, V. Kolmogorov, and A. Blake. "grabcut": interactive foreground extraction using iterated graph cuts. In ACM SIGGRAPH 2004 Papers, SIGGRAPH '04, pages 309--314, New York, NY, USA, 2004. ACM.

Digital Library

[29]

A. Schödl, R. Szeliski, D. Salesin, and I. Essa. Video textures. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 489--498, New York, New York, USA, 2000. ACM Press/Addison-Wesley Publishing Co.

Digital Library

[30]

P. M. A. Sousa and M. J. Fonseca. Geometric matching for clip-art drawing retrieval. J. Visual Communication and Image Representation, 20(2):71--83, 2009.

Digital Library

[31]

R. Vezanni, D. Baltieri, and R. Cucchiara. HMM based action recognition with projection histogram features. In Proc. Intl. Conf. Pattern Recognition (ICPR), volume 6388, pages 290--297, 2010.

[32]

Y. Yang and D. Ramanan. Articulated pose estimation using flexible mixtures of parts. In Proc. Computer Vision and Pattern Recognition, 2011.

Digital Library

Cited By

Yoshida NLiu J(2022)Highly Robust Action Retrieval using View-invariant Pose Feature and Simple yet Effective Query Expansion Method2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.23919/APSIPAASC55919.2022.9979865(1269-1277)Online publication date: 7-Nov-2022
https://doi.org/10.23919/APSIPAASC55919.2022.9979865
Yoshida NLiu JShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)On-demand Action Detection System using Pose InformationProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3478567(2810-2812)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3478567
Jürgens SCorreia NMasu RKaltenbrunner MMurer MWolf KOakley I(2021)The Body Beyond Movement: (Missed) Opportunities to Engage with Contemporary Dance in HCIProceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3430524.3440624(1-9)Online publication date: 14-Feb-2021
https://dl.acm.org/doi/10.1145/3430524.3440624
Show More Cited By

Index Terms

ReEnact: Sketch based Choreographic Design from Archival Dance Footage
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Information systems
  1. Information systems applications

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMR '14: Proceedings of International Conference on Multimedia Retrieval

April 2014

564 pages

ISBN:9781450327824

DOI:10.1145/2578726

Conference Chairs:
Mohan Kankanhalli
National University of Singapore
,
Stefan Rueger
The Open University, UK
,
R. Manmatha
A9.com, USA
,
General Chairs:
Joemon Jose
University of Glasgow, UK
,
Keith van Rijsbergen
University of Glasgow, UK

Copyright © 2014 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

In-Cooperation

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2014

Check for updates

Qualifiers

Tutorial
Research
Refereed limited

Conference

ICMR '14

ICMR '14: International Conference on Multimedia Retrieval

April 1 - 4, 2014

Glasgow, United Kingdom

Acceptance Rates

ICMR '14 Paper Acceptance Rate 21 of 111 submissions, 19%;

Overall Acceptance Rate 254 of 830 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yoshida NLiu J(2022)Highly Robust Action Retrieval using View-invariant Pose Feature and Simple yet Effective Query Expansion Method2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.23919/APSIPAASC55919.2022.9979865(1269-1277)Online publication date: 7-Nov-2022
https://doi.org/10.23919/APSIPAASC55919.2022.9979865
Yoshida NLiu JShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)On-demand Action Detection System using Pose InformationProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3478567(2810-2812)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3478567
Jürgens SCorreia NMasu RKaltenbrunner MMurer MWolf KOakley I(2021)The Body Beyond Movement: (Missed) Opportunities to Engage with Contemporary Dance in HCIProceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3430524.3440624(1-9)Online publication date: 14-Feb-2021
https://dl.acm.org/doi/10.1145/3430524.3440624
Yu QSong JSong YXiang THospedales T(2021)Fine-Grained Instance-Level Sketch-Based Image RetrievalInternational Journal of Computer Vision10.1007/s11263-020-01382-3129:2(484-500)Online publication date: 1-Feb-2021
https://dl.acm.org/doi/10.1007/s11263-020-01382-3
Ho TVirtusio JChen YHsu CHua K(2020)Sketch-guided Deep Portrait GenerationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/339623716:3(1-18)Online publication date: 5-Jul-2020
https://dl.acm.org/doi/10.1145/3396237
Dey SRiba PDutta ALlados JSong Y(2019)Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2019.00228(2174-2183)Online publication date: Jun-2019
https://doi.org/10.1109/CVPR.2019.00228
Chen WHays J(2018)SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition10.1109/CVPR.2018.00981(9416-9425)Online publication date: Jun-2018
https://doi.org/10.1109/CVPR.2018.00981
Aich AMallick TBhuyan HDas PMajumdar A(2018)NrityaGuru: A Dance Tutoring System for Bharatanatyam Using KinectComputer Vision, Pattern Recognition, Image Processing, and Graphics10.1007/978-981-13-0020-2_42(481-493)Online publication date: 26-Apr-2018
https://doi.org/10.1007/978-981-13-0020-2_42
Ribeiro CKuffner RFernandes C(2018)Virtual Reality Annotator: A Tool to Annotate Dancers in a Virtual EnvironmentDigital Cultural Heritage10.1007/978-3-319-75826-8_21(257-266)Online publication date: 23-May-2018
https://doi.org/10.1007/978-3-319-75826-8_21
Yelamarthi SReddy SMishra AMittal A(2018)A Zero-Shot Framework for Sketch Based Image RetrievalComputer Vision – ECCV 201810.1007/978-3-030-01225-0_19(316-333)Online publication date: 8-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-030-01225-0_19
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents