Abstract
In this paper, we present a simple and robust mixed reality (MR) framework that allows for real-time interaction with virtual humans in mixed reality environments under consistent illumination. We will look at three crucial parts of this system: interaction, animation and global illumination of virtual humans for an integrated and enhanced presence. The interaction system comprises of a dialogue module, which is interfaced with a speech recognition and synthesis system. Next to speech output, the dialogue system generates face and body motions, which are in turn managed by the virtual human animation layer. Our fast animation engine can handle various types of motions, such as normal key-frame animations, or motions that are generated on-the-fly by adapting previously recorded clips. Real-time idle motions are an example of the latter category. All these different motions are generated and blended on-line, resulting in a flexible and realistic animation. Our robust rendering method operates in accordance with the previous animation layer, based on an extended for virtual humans precomputed radiance transfer (PRT) illumination model, resulting in a realistic rendition of such interactive virtual characters in mixed reality environments. Finally, we present a scenario that illustrates the interplay and application of our methods, glued under a unique framework for presence and interaction in MR.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Alice chat bot: http://www.alicebot.org/. Cited November (2005)
Amanatides, J., Woo, A.: A fast voxel traversal algorithm for ray tracing. In: G. Marechal (ed.) Proceedings of Eurographics ’87, pp. 3–10. Elsevier, Amsterdam (1987)
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances in augmented reality. IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)
Balcisoy, S.S.: Analysis and development of interaction techniques between real and synthetic worlds. Dissertation, EPFL (2001)
Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proceedings of SIGGRAPH ’01, pp. 477–486 (2001)
Cavazza, M., Martin, O., Charles, F., Mead, S., Marichal, X.: Users acting in mixed reality interactive storytelling. In: Proceedings of 2nd International Conference on Virtual Storytelling, pp. 189–197 (2003)
Cohen, M.M., Massaro, D.W.: Modeling coarticulation in synthetic visual speech. In: N. Magnenat-Thalmann, D. Thalmann (eds.) Models and Techniques in Computer Animation, pp. 139–156. Springer, Berlin Heidelberg New York (1993)
Egges, A., Kshirsagar, S., Magnenat-Thalmann, N.: Generic personality and emotion simulation for conversational agents. Comput. Anim. Virt. Worlds 15(1), 1–13 (2004)
Egges, A., Magnenat-Thalmann, N.: Emotional communicative body animation for multiple characters. In: Proceedings of the 1st International Workshop on Crowd Simulation (V-Crowds), pp. 31–40 (2005)
Egges, A., Molet, T., Magnenat-Thalmann, N.: Personalised real-time idle motion synthesis. In: Pacific Graphics 2004, pp. 121–130 (2004)
Garchery, S.: Animation faciale temps-reel multi plates-formes. Dissertation, MIRALab, University of Geneva (2004)
Grassia, F.S.: Practical parameterization of rotations using the exponential map. J. Graph. Tools 3(3), 29–48 (1998)
H-Anim Humanoid Animation Working Group: Specification for a standard humanoid. http://www.h-anim.org/. Cited May (2006)
Hartmann, B., Mancini, M., Pelachaud, C.: Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis. In: Computer Animation 2002, pp. 111–119 (2002)
Inui, T., Tanabe, Y., Onodera, Y.: Group Theory and its Applications in Physics. Springer, Berlin Heidelberg New York (1990)
Ivanic, J., Ruedenberg, K.: Additions and corrections: rotation matrices for real spherical harmonics. J. Phys. Chem. 102(45), 9099–9100 (1998)
Kautz, J., Lehtinen, J., Sloan, P.P.: Precomputed radiance transfer: theory and practise. In: ACM SIGGRAPH ’05 Course Notes (2005)
Kopp, S., Wachsmuth, I.: Synthesizing multimodal utterances for conversational agents. Comput. Anim. Virt. Worlds 15(1), 39–52 (2004)
Kovar, L., Gleicher, M.: Flexible automatic motion blending with registration curves. In: Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 214–224 (2003)
Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. In: Proceedings of SIGGRAPH ’02, pp. 473–482 (2002)
Krenn, B., Pirker, H.: Defining the gesticon: language and gesture coordination for interacting embodied agents. In: Proceedings of the AISB-2004 Symposium on Language, Speech and Gesture for Expressive Characters, pp. 107–115. University of Leeds, UK (2004)
Kshirsagar, S., Molet, T., Magnenat-Thalmann, N.: Principal components of expressive speech animation. In: Computer Graphics International 2001, pp. 38–44. IEEE Press, Washington, DC (2001)
Microsoft Speech SDK version 5.1 (SAPI5.1): http://www.microsoft.com/speech/download/sdk51/. Cited May (2006)
Mueller, M., Roeder, T., Clausen, M.: Efficient content-based retrieval of motion capture data. In: Proceedings SIGGRAPH ’05, pp. 677–685 (2005)
Openscenegraph: http://www.openscenegraph.org/. Cited May (2006)
Papagiannakis, G., Foni, A., Magnenat-Thalmann, N.: Practical precomputed radiance transfer for mixed reality. In: Proceedings of Virtual Systems and Multimedia 2005, pp. 189–199. VSMM Society, Yanagido, Japan (2005)
Papagiannakis, G., Kim, H., Magnenat-Thalmann, N.: Believability and presence in mobile mixed reality environments. In: IEEE VR2005 Workshop on Virtuality Structures (2005)
Perlin, K.: An image synthesizer. In: Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, pp. 287–296. ACM, Boston (1985)
Perlin, K.: Real time responsive animation with personality. IEEE Trans. Visual. Comput. Graph. 1(1), 5–15 (1995)
Poggi, I., Pelachaud, C., Rosis, F.D., Carofiglio, V., Carolis, B.D.: Greta: a believable embodied conversational agent. In: O. Stock, M. Zancanaro (eds.) Multimodal Intelligent Information Presentation, vol. 27. Springer, Berlin Heidelberg New York (2005)
Ponder, M., Papagiannakis, G., Molet, T., Magnenat-Thalmann, N., Thalmann, D.: Vhd++ development framework: towards extendible, component based VR/AR simulation engine featuring advanced virtual character technologies. In: Proceedings of Computer Graphics International (CGI), pp. 96–104. IEEE Press, Washington, DC (2003)
Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of SIGGRAPH ’01. ACM, Boston (2001)
Ren, Z., Wang, R., Snyder, J., Zhou, K., Liu, X., Sun, B., Sloan, P.P., Bao, H., Peng, Q., Guo, B.: Real-time soft shadows in dynamic scenes using spherical harmonic exponentiation. In: Prococeedings of ACM SIGGRAPH ’06, pp. 977–986 (2006)
Rose, C., Cohen, M., Bodenheimer, B.: Verbs and adverbs: multidimensional motion interpolation. IEEE Comput. Graph. Appl. 18(5), 32–48 (1998)
Rose, C., Guenter, B., Bodenheimer, B., Cohen, M.: Efficient generation of motion transitions using spacetime constraints. In: Proceedings of ACM SIGGRAPH ’96, Annual Conference Series, pp. 147–154 (1996)
Sloan, P.P., Kautz, J., Snyder, J.: Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In: Proceedings of ACM SIGGRAPH ’02, pp. 527–536. ACM, Boston (2002)
Tamura, H.: Mixed reality: future dreams seen at the border between real and virtual worlds. IEEE Comput. Graph. Appl. 21(6), 64–70 (2001)
Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., Piekarski, W.: Arquake: an outdoor/indoor augmented reality first person application. In: Proceedings of Symposium on Wearable Computers, pp. 139–146 (2000)
Unuma, M., Anjyo, K., Tekeuchi, T.: Fourier principles for emotion-based human figure animation. In: Proceedings of ACM SIGGRAPH ’95, Annual Conference Series, pp. 91–96 (1995)
Virtual Human Markup Language (vhml):http://www.vhml.org/. Cited November (2005)
Vacchetti, L., Lepetit, V., Ponder, M., Papagiannakis, G., Fua, P., Thalmann, D., Magnenat-Thalmann, N.: Stable real-time AR framework for training and planning in industrial environments. In: S.K. Ong, A.Y.C. Nee (eds.) Virtual Reality and Augmented Reality Applications in Manufacturing. Springer, Berlin Heidelberg New York (2004)
Vlahakis, V., Ioannidis, N., Karigiannis, J., Tsotros, M., Gounaris, M., Stricker, D., Gleue, T., Daehne, P., Almeida, L.: Archeoguide: an augmented reality guide for archaeological sites. IEEE Comput. Graph. Appl. 22(5), 52–60 (2002)
Wiley, D., Hahn, J.: Interpolation synthesis of articulated figure motion. IEEE Comput. Graph. Appl. 17(6), 39–45 (1997)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License ( https://creativecommons.org/licenses/by-nc/2.0 ), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Egges, A., Papagiannakis, G. & Magnenat-Thalmann, N. Presence and interaction in mixed reality environments. Visual Comput 23, 317–333 (2007). https://doi.org/10.1007/s00371-007-0113-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-007-0113-z