research-article

Mutual text-image queries

Authors:

Timo Götzelmann,

Pere-Pau Vázquez,

Andreas Nürnberger,

Thomas StrothotteAuthors Info & Claims

SCCG '07: Proceedings of the 23rd Spring Conference on Computer Graphics

Pages 139 - 146

https://doi.org/10.1145/2614348.2614368

Published: 26 April 2007 Publication History

Abstract

This paper presents a novel approach to support students to learn a comprehensive domain-specific terminology and to understand textual descriptions of complex-shaped objects. We implemented an experimental system where learners can interactively explore textual descriptions and 3D visualizations. We propose a method for hierarchical content representations of text documents and views on 3D models. Based on these data structures, user interactions on texts and interactive 3D visualizations are transformed into queries to an information retrieval system. This enables us to coordinate the content of both media, to focus the attention of the user on the most salient graphical objects, and to suggest potential relevant text segments in large text documents and appropriate views on 3D models to illustrate the spatial relations between the relevant domain objects of the query. Finally, we demonstrated this concept in an interactive tutoring environment based on standard textbooks on human anatomy.

References

[1]

Bruckner, S., and Gröller, E. 2005. VolumeShop: An Interactive System for Direct Volume Illustrations. In IEEE Visualization, 671--678.

[2]

Carpendale, M. S. T., and Montagnese, C. 2001. A Framework for Unifying Presentation Space. In 14th Ann. ACM Symp. on User Interface Software and Technology, 61--70.

Digital Library

[3]

Collins, A. M., and Loftus, E. F. 1975. A Spreading-Activation Theory of Semantic Processing. Psychological Review 82, 6, 407--428.

[4]

Cutting, D. et al., 2007. Lucene text retrieval engine.

[5]

Diepstraten, J., Weiskopf, D., and Ertl, T. 2002. Transparency in Interactive Technical Illustrations. Computer Graphics Forum 21, 3, 317--325.

[6]

Digimation Inc., 2007. Viewpoint 3D Library.

[7]

Feiner, S. K., and McKeown, K. R. 1993. Automating the Generation of Coordinated Multimedia Explanations. In Intelligent Multimedia Interfaces, M. T. Maybury, Ed. AAAI Press, Menlo Park, 117--138.

Digital Library

[8]

Furnas, G. W. 1986. Generalized Fisheye Views. In Conf. on Human Factors in Computing Systems, 16--23.

Digital Library

[9]

Germer, T., Götzelmann, T., Spindler, M., and Strothotte, T. 2006. SpringLens: Distributed Nonlinear Magnifications. In Eurographics'06 Short Papers, 123--126.

[10]

Gray, H. 1918. Anatomy of the Human Body, 20th ed. Lea & Febiger, Philadelphia.

[11]

Götze, M., Neumann, P., and Isenberg, T. 2005. User-Supported Interactive Illustration of Text. In Simulation und Visualisierung, 195--206.

[12]

Götzelmann, T., Hartmann, K., and Strothotte, T. 2006. Agents-Based Annotation of Interactive 3D Visualizations. In 6th Int. Symp. on Smart Graphics, 24--35.

[13]

Götzelmann, T., Götze, M., Ali, K. Hartmann, K., and Strothotte, T. 2007. Annotating Images through Adaptation: An Integrated Text Authoring and Illustration Framework. Journal of the WSCG 15, 1.

[14]

Götzelmann, T., Vázquez, P.-P., Hartmann, K., Nürnberger, A., and Strothotte, T. 2007. Correlating Text and Images: Concept and Evaluation. In 7th Int. Symp. on Smart Graphics. (accepted).

Digital Library

[15]

Hartmann, K., and Strothotte, T. 2002. A Spreading Activation Approach to Text Illustration. In 2nd Int. Symp. on Smart Graphics, 39--46.

Digital Library

[16]

Kamada, T., and Kawai, S. 1988. A Simple Method for Computing General Position in Displaying Three-Dimensional Objects. Computer Vision, Graphics, and Image Processing 41, 1, 43--56.

Digital Library

[17]

Keahey, T. A., and Robertson, E. L. 1997. Nonlinear Magnification Fields. In IEEE Symp. on Information Visualization, 51.

Digital Library

[18]

Lieberman, H., Liu, H., Singh, P., and Barry, B. 2005. Beating Common Sense Into Interactive Applications. AI Magazine 25, 4 (Winter), 63--76.

[19]

National Library of Medicine, 2007. Visible Human Project.

[20]

Plemenos, D., and Benayada, M. 1996. Intelligent Display in Scene Modeling. New Techniques to Automatically Compute Good Views. In Int. Conf. on Computer Graphics & Vision.

[21]

Salton, G., Wong, A., and Yang, C. S. 1975. A Vector Space Model for Automatic Indexing. Communications of the ACM 18, 11, 613--620.

Digital Library

[22]

Salton, G., Allan, J., Buckley, C., and Singhal, A. 1994. Automatic Analysis, Theme Generation, and Summarization of Machine-Readable Texts. Science 264, 1421--1426.

[23]

Schlechtweg, S., and Strothotte, T. 1999. Illustrative Browsing: A New Method of Browsing in Long On-line Texts. In Int. Conf. on Computer Human Interaction, 466--473.

[24]

Schlechtweg, S., and Strothotte, T. 2000. Generating Scientific Illustrations in Digital Books. In AAAI Spring Symp. on Smart Graphics, 8--15.

[25]

Shannon, E. C. 1948. A Mathematical Theory of Communication. The Bell System Technical Journal 27 (July--October), 379--423,623--656.

[26]

Tortora, G. J. 1997. Introduction to the Human Body: The Essentials of Anatomy and Psychology. Benjamin Cummings, Menlo Park, CA.

[27]

Viola, I., Kanitsar, A., and Gröller, M. E. 2004. Importance-Driven Volume Rendering. In IEEE Visualization 2004, 139--145.

Digital Library

[28]

Viola, I., Feixas, M., Sbert, M., and Gröller, M. E. 2006. Importance-Driven Focus of Attention. IEEE Transactions on Visualization and Computer Graphics 12, 5.

Digital Library

[29]

Vázquez, P.-P., Feixas, M., Sbert, M., and Heidrich, W. 2001. Viewpoint Selection using Viewpoint Entropy. In Vision Modeling and Visualization Conference, 273--280.

Digital Library

[30]

Vázquez, P. P., Feixas, M., Sbert, M., and Llobet, A. 2006. Realtime Automatic Selection of Good Molecular Views. Computers & Graphics 30, 1, 98--110.

Digital Library

[31]

Wahlster, W., André, E., Finkler, W., Profitlich, H.-J., and Rist, T. 1993. Plan-Based Integration of Natural Language and Graphics Generation. Artificial Intelligence 63, 387--427.

Digital Library

[32]

Yang, Y., Chen, J. X., and Beheshti, M. 2005. Nonlinear Perspective Projections and Magic Lenses: 3D View Deformation. Computer Graphics & Applications 25, 1, 76--84.

Digital Library

Cited By

Bettio FAhsan MMarton FGobbetti E(2021)A novel approach for exploring annotated data with interactive lensesComputer Graphics Forum10.1111/cgf.1431540:3(387-398)Online publication date: 29-Jun-2021
https://doi.org/10.1111/cgf.14315
Rodriguez MAgus MMarton FGobbetti E(2015)Adaptive Recommendations for Enhanced Non-linear Exploration of Annotated 3D ObjectsComputer Graphics Forum10.5555/2858877.285888334:3(41-50)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.5555/2858877.2858883
Kohlmann PBruckner SKanitsar AGroller M(2009)Contextual picking of volumetric structuresProceedings of the 2009 IEEE Pacific Visualization Symposium10.1109/PACIFICVIS.2009.4906855(185-192)Online publication date: 20-Apr-2009
https://dl.acm.org/doi/10.1109/PACIFICVIS.2009.4906855
Show More Cited By

Index Terms

Mutual text-image queries

Recommendations

Image abstraction in crossmedia retrieval for text illustration
ECIR'12: Proceedings of the 34th European conference on Advances in Information Retrieval

Text illustration is a multimedia retrieval task that consists in finding suitable images to illustrate text fragments such as blog entries, news reports or children stories. In this paper we describe a crossmedia retrieval system which, given a textual ...
Some results using different approaches to merge visual and text-based features in CLEF'08 photo collection
CLEF'08: Proceedings of the 9th Cross-language evaluation forum conference on Evaluating systems for multilingual and multimodal information access

This paper describes the participation of the MIRACLE team at the ImageCLEF Photographic Retrieval task of CLEF 2008. We succeeded in submitting 41 runs. Obtained results from text-based retrieval are better than content-based as previous experiments in ...
Towards Text Search for Information Visualization Retrieval
CHI EA '15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems

We investigate the use of text search to retrieve information visualizations. This is important as the body of available visualizations on the web is growing. And it the meantime, they are difficult to find as they don't have an immediate textual ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SCCG '07: Proceedings of the 23rd Spring Conference on Computer Graphics

April 2007

242 pages

ISBN:9781605589565

DOI:10.1145/2614348

Conference Chair:
Mateu Sbert
University of Girona
,
Editor:
Stephen Spencer
University of Washington

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Comenius University: Comenius University

In-Cooperation

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques
EUROGRAPHICS: The European Association for Computer Graphics

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SCCG07

Sponsor:

Comenius University

SCCG07: Spring Conference on Computer Graphics

April 26 - 28, 2007

Budmerice, Slovakia

Acceptance Rates

Overall Acceptance Rate 67 of 115 submissions, 58%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
33
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bettio FAhsan MMarton FGobbetti E(2021)A novel approach for exploring annotated data with interactive lensesComputer Graphics Forum10.1111/cgf.1431540:3(387-398)Online publication date: 29-Jun-2021
https://doi.org/10.1111/cgf.14315
Rodriguez MAgus MMarton FGobbetti E(2015)Adaptive Recommendations for Enhanced Non-linear Exploration of Annotated 3D ObjectsComputer Graphics Forum10.5555/2858877.285888334:3(41-50)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.5555/2858877.2858883
Kohlmann PBruckner SKanitsar AGroller M(2009)Contextual picking of volumetric structuresProceedings of the 2009 IEEE Pacific Visualization Symposium10.1109/PACIFICVIS.2009.4906855(185-192)Online publication date: 20-Apr-2009
https://dl.acm.org/doi/10.1109/PACIFICVIS.2009.4906855
Götzelmann TVázquez PHartmann KNürnberger AStrothotte T(2007)Correlating Text and ImagesProceedings of the 8th international symposium on Smart Graphics10.1007/978-3-540-73214-3_9(97-109)Online publication date: 25-Jun-2007
https://dl.acm.org/doi/10.1007/978-3-540-73214-3_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents