PoCoMo: Projected Collaboration using Mobile Devices

The MIT Faculty has made this article openly available. Please share
how this access benefits you. Your story matters.

Citation Roy Shilkrot, Seth Hunter, and Patricia Maes. 2011. PoCoMo:
projected collaboration using mobile devices. In Proceedings of
the 13th International Conference on Human Computer
Interaction with Mobile Devices and Services (MobileHCI '11).
ACM, New York, NY, USA, 333-336
As Published http://dx.doi.org/10.1145/2037373.2037424
Publisher Association for Computing Machinery (ACM)

Version Author's final manuscript

Accessed Tue Jan 02 11:12:49 EST 2018
Citable Link http://hdl.handle.net/1721.1/80404
Terms of Use Creative Commons Attribution-Noncommercial-Share Alike 3.0
Detailed Terms http://creativecommons.org/licenses/by-nc-sa/3.0/
 PoCoMo: Projected Collaboration using Mobile Devices
Roy Shilkrot Seth Hunter Patricia Maes
MIT Media Lab MIT Media Lab MIT Media Lab
75 Amherst St., Cambridge, MA 75 Amherst St., Cambridge, MA 75 Amherst St., Cambridge, MA
roys@media.mit.edu hunters@media.mit.edu pattie@media.mit.edu
ABSTRACT systems, embedded in fully functional mobile phones. The
As personal projection devices become more common they proposed system’s goal is to enable users to interact with
will be able to support a range of exciting and unexplored one another and their environment, in a natural, ad-hoc
social applications. We present a novel system and method way. The phones display projected characters that align
that enables playful social interactions between multiple with the view of the camera on the phone, which is running
projected characters. The prototype consists of two mobile software that tracks the position of the characters. Two
projector-camera systems, with lightly modified existing characters projected in the same area will quickly respond
hardware, and computer vision algorithms to support a to one another in context of the features of the local
selection of applications and example scenarios. Our system environment.
allows participants to discover the characteristics and
behaviors of other characters projected in the environment. PoCoMo is implemented on commercially available
The characters are guided by hand movements, and can hardware and standardized software libraries to increase the
respond to objects and other characters, to simulate a mixed likelihood of implementation by developers and to provide
reality of life-like entities. example scenarios, which advocate for projector-enabled
mobile phones. Multiple phone manufacturers have
Author Keywords experimental models with projection capabilities, but to our
Mobile projection. Collaborative play. Social interaction. knowledge none of them have placed the camera in line of
Mixed-reality characters. Animaiton. Augmented reality. sight of the projection area. We expect models that support
this capability for AR purposes to become available in the
ACM Classification Keywords near future.
I.5.4 Applications: Computer vision, I.2.10 Vision and
Scene Understanding: Video analysis, K.8 Personal
Computing: Games, H.5.3 Group and Organization
Interfaces: Synchronous Interaction, H.5.2 User Interfaces:
Input Devices and Strategies.

General Terms
Algorithms, Design, Experimentation, Human Factors

INTRODUCTION (a) (b)

The use of mobile devices for social and playful
interactions is a rapidly growing field. Mobile phones now Figure 1. (a) Users hold micro projector-camera devices to
provide the high-performance computing power that is project animated characters on the wall. (b) The characters
recognize and interact with one another.
needed to support high-resolution animation via embedded
graphics processors, and an always-on Internet connectivity
via 3G cellular networks. Further advances in mobile PoCoMo makes use of a standard object tracking technique
projection hardware make possible the exploration of more [5] to sense the presence of other projected characters, and
seamless social interactions between co-located users with extract visual features from the environment as part of a
mobile phones. PoCoMo is a system developed in our lab to game. The algorithm is lightweight, enabling operation in a
begin prototyping social scenarios enabled by collaborative limited resource environment, without a great decrease in
mobile projections in the same physical environment. usability.
PoCoMo is a set of mobile, handheld projector-camera Extending mobile phones to externalize display information
in the local environment and respond to other players
creates new possibilities for mixed reality interactions that
Permission to make digital or hard copies of all or part of this work for are relatively unexplored. We focus in particular on the
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies social scenarios of acknowledgment, relating, and exchange
bear this notice and the full citation on the first page. To copy otherwise, in games enabled between two co-located users. This paper
or republish, to post on servers or to redistribute to lists, requires prior describes the design of the system, implementation details,
specific permission and/or a fee. and our initial usage scenarios.
MobileHCI 2011, Aug 30–Sept 2, 2011, Stockholm, Sweden.
Copyright 2011 ACM 978-1-4503-0541-9/11/08-09....$10.00.
RELATED WORK
The increasing portability of projection systems since 2003
has enabled researchers to explore geometrically aware
collaborative systems based on orientation. ILamps and
RFIG [10] by Raskar et al. examined the geometric issues
necessary for adaptive images of multiple projectors into a
single display. The Hotaru system [11] uses static
projection to simulate multiple projections. Several
researchers have introduced the flashlight metaphor
[1,2,6,9,12] for interacting with the physical world, where a
portion of the virtual world is revealed by the projection.
Figure 2. Character rigs, and component parts for
Cao et al. [3] presented techniques for revealing contextual animation.
information on the fly also using the flashlight metaphor.
Ruzkio and Greaves [6] present a body of work exploring of friendship such as waving (medium distance) or
interaction using personal projection, however they mostly shaking hands (close proximity).
rely on an external 6-DOF tracking system, and a single-
• Mutual Exploration When characters are moved through
user paradigm.
the environment they begin a walk cycle, which is
Other researchers have explored playful and game-like correspondent the speed of change of position. If the
possibilities enabled by portable projection technologies. characters are within view of each other, their cycles
Twinkle [14] by Yoshida et al., describes a handheld approximately match. The synchronized steps help create
projector-camera system projecting a character that the feeling of a shared environment, and provide
interacts with drawings on a whiteboard. Their system feedback about proximity.
specifically reacts to edges, either drawn or created by • Leave-a-Present Characters can find and leave presents
objects it the scene but does not respond other systems in to one another at distinct features in the environment. We
the same area. Recently, Willis et al. introduced provide this as an example of collaborative interactions
MotionBeam [12], a technique for projected character that can extend into multiple sessions and relate to
control via handheld devices. VisiCon [8] and CoGAME specific physical locations.
[7], are collaborative robot-controlling games using
handheld projectors. In this work the camera is mounted on Detection and Tracking Methods
the robot instead of being embedded in the mobile phone. In our system, each device projects a character that should
be detected and tracked, both by the device itself and other
SYSTEM OVERVIEW participating devices in the same projection area. For this
PoCoMo is a compact mobile projector-camera system, purpose we implemented a simple marker-based tracking
built on a commercially available cellular smartphone that algorithm, inspired by the seminal work on Active
includes a pico-projector and video camera. In the current Appearance Models [4] (AAM) and color-based tracking
prototype, the projector and camera are aligned in the same methods [5,13]. Each character has an identity that is
field of view via a plastic encasing and a small mirror (see represented by the color of its markers. For instance one
Figure 1). The setup enables the program to track both the character may have blue markers, while the other character
projected image and a portion of the surrounding surface. may have red markers. The detection algorithm scans the
All algorithms are executed locally on the phones. image for the circular markers by first applying a threshold
The characters are programmed to have component parts on the Hue, Saturation and Value, extracting the contour
(see Figure 2) with separate articulation and trigger poly-lines. Following the work on AAM, we create a
different sequences based on the proximity of the other marker of a predefined shape, e.g. a circle, and match its
characters in order to appear to be life-like in the contour to the blobs discovered in the masking stage. The
environment. We have designed characters for a number of two highest-ranking regions are considered to be the
scenarios to demonstrate how such a system may be used to markers. In the following iterations, similar to [5], we
for playful collaborative experiences: consider the distance of the region to the previous detected
markers as a weight to the rank equation:
• Character Acknowledgment The projected characters
can respond to the presence and orientation of one w ( R) = Rarea " d Rcenter , Rˆ center " e( Rcontour )
( )
another and express their interest in interaction. Initially
they acknowledge each other by turning to face one where R is the candidate region, d is a simple L2 distance
another and smiling. function and e is the ellipse-matching measure.
• Icebreaker Interactions When characters have The red character will look for blue markers, and vice
!
acknowledged each other, a slow approach by one versa, however both characters will also look for their own
character to the proximity to another can trigger a gesture markers for calibration purposes.
 to index the key frame timing. The second character reads
the signal, and initializes their sequence as well. The
handshake sequences of the characters are designed to have
equal length animations for interoperability.
Leave-a-Present. The characters can also leave a present on
a certain object for the other character to find. This is done
when the character is placed near an object, without
movement, for over 3 seconds. An animation starts,
showing the character leaning to leave a box on the ground,
Figure 3. Rectification of the characters using the projected and the user is prompted for an object to leave: an image or
markers, under the assumption that the projectors and
any kind of file on the phone. The system extracts SURF
cameras are positioned perpendicular to the projection
plane. The relation between dRED and dBLUE and the angle !,
descriptors of key-points on the object in the scene, and
determine the rigid transformation applied to the red sends the data along with the batch of descriptors to a
character. central server. The present location is recorded relative to
strong features like corners and high contrast edges. When a
The characters have markers on the opposite corners of the character is in present-seeking mode, it tries to look for
projection, i.e. top-right and bottom-left. Under the features in the scene that match those in the server. If a
assumption of having both the camera plane and projector feature match is strongly correlated, the present will begin
parallel to the projection plane this layout allows to fade in. We assume that users will leave presents in
understanding the rotation angle and size of the projected places they have previously interacted, to provide a context
character. These assumptions make the detection far for searching for any virtual items in the scene.
quicker, in comparison to assuming arbitrary projective
transformations from the projection plane, while Gestures
maintaining a lively experience. It also allows each system Our system is responsive to the phone’s motion sensing
to determine the relation between the self-character and capabilities, an accelerometer and gyroscope. The
friend character, so the projection size and rotation can be characters can begin a walk sequence animation when the
adjusted to match, as Figure 3 shows. To maximize device is detected to move in a certain direction at a
resolution, the larger projected character will try to match constant rate. The current accelerometer technology used in
its size and rotation to the smaller one. our prototype only gives out a derivative motion reading,
We are limited by the brightness (12 lumens) of our current i.e. on the beginning and end of the motion itself and not
projectors and therefore the scale and size of our current during. Therefore we use two cues: Start of Motion and End
projected markers are exaggerated. Work by Costanza et al. of Motion, which trigger the starting and halting of the walk
[5] emphasizes the importance of incorporating the design sequence. The system also responds to shaking gestures,
of the markers with the visual content. In future work we which breaks the current operation of the character.
would like to incorporate the markers into the design of the
characters, and use them to provide feedback about the state Hardware
of the character to the user. We chose to use the Samsung Halo projector-phone as the
platform for our system. This Android-OS system has a
IMPLEMENTATION DETAILS AND CODE versatile development environment. The tracking and
detection algorithms were implemented in C++ and
Character Interaction compiled to a native library and the UI of the application
Following Head. In the initial sensing stage, the projected was implemented in Java using Android API. We used the
character displays an interested look: The eyes follow the OpenCV library to implement the computer vision
possible location of the other character. The presence of algorithms.
another character is determined once the two markers were
observed in the scene for over 20 frames in the duration of The Prototype. To align the camera with the projection, we
the last 40 frames. Once the presence of another character designed a casing with a mirror that fits on the device.
in view is the operated character will face the friend Figure 4 shows the development stages in the design of the
character. If the characters linger for 80-100 frames they prototype. First, out prototype was attached to the device,
will wave in the direction of the other. If the characters are and then we transitioned into full-device casing to support
aligned they will attempt to do a handshake. Alignment is handling the phone. Finally, we designed a rounded case
considered sufficient if the center point of the friend made from a more robust plastic to protect the mirror and
character projection is within a normalized range. allow it to be decoupled from the phone.
Handshake. The handshake is a key-frame animation of 10 FUTURE WORK
frames, with 10 frame transitions from a standing pose. In In future systems we plan to make the detection more
order to synchronize the animation the initiating characters robust and integrate markers with the projected content. We
displays a third colored marker for a small period of time, also plan to develop animations that correlate to specific
user profiles and migrate the application to devices with
 User interface software and technology, ACM
(2007), 43–52.
4. Cootes, T.F., Edwards, G.J., and Taylor, C.J. Active
appearance models. IEEE Transactions on Pattern
Analysis and Machine Intelligence 23, 6 (2001),
(a) v1.0 (b) v2.0 (c) v3.0 681-685.
5. Fieguth, P. and Terzopoulos, D. Color-based
tracking of heads and other mobile objects at video
frame rates. Proceedings of IEEE Computer Society
Conference on Computer Vision and Pattern
Recognition, , 21-27.
(d) v0.9 (e) v2.0 Sim. (f) v3.0 Sim.
Figure 4. Development stages of the phone case prototype. (e,g) 6. Greaves, A. and Rukzio, E. Interactive Phone Call:
3D rendering of the prototypes design. Exploring Interactions during Phone Calls using
Projector Phones. Workshop on Ensembles of On-
Body Devices at Mobile HCI, (2010).
wider fields of views for the projector and camera. In
addition, we will conduct a user study to assess the fluidity 7. Hosoi, K., Dao, V.N., Mori, A., and Sugimoto, M.
of the interaction and improve the scenarios. CoGAME: manipulation using a handheld
projector. ACM SIGGRAPH 2007 emerging
CONCLUSION technologies, ACM (2007), 2.
We present PoCoMo, a system for collaborative mobile
interactions with projected characters. PoCoMo is an 8. Hosoi, K., Dao, V.N., Mori, A., and Sugimoto, M.
integrated mobile platform independent of external devices VisiCon : A Robot Control Interface for Visualizing
and physical markers. The proposed methods do not require Manipulation Using a Handheld Projector
any setup and can be played in any location. Cooperative Robot Navigation Game via
Projection. Human Factors, (2007), 99-106.
Our system allows users to interact together in the same
physical environment with animated mixed-reality 9. Rapp, S., Michelitsch, G., Osen, M., et al. Spotlight
characters. The use of projectors for multiplayer gaming Navigation: Interaction with a handheld projection
affords more natural and spontaneous shared experiences. device. Advances in Pervasive Computing: A
Closely related systems support utilitarian tasks such as collection of Contributions Presented at
sharing [3] or inventory management [10]. We explore PERVASIVE, (2004), 397-400.
novel social interactions enabled by mobile projectors and 10. Raskar, R., Baar, J. van, Beardsley, P., Willwacher,
hope developers will contribute other scenarios and games T., Rao, S., and Forlines, C. iLamps: geometrically
within the domain of collaborative mobile projection. aware and self-configuring projectors. ACM
SIGGRAPH 2005 Courses, ACM (2005), 5.
ACKNOWLEDGMENTS
We would like to thank Amit Zoran for help with the 11. Sugimoto, M., Miyahara, K., Inoue, H., and
prototype design, and the Fluid Interfaces group members Tsunesada, Y. Hotaru: intuitive manipulation
for their critique and insightful remarks. techniques for projected displays of mobile devices.
Human-Computer Interaction-INTERACT 2005,
REFERENCES (2005), 57-68.
1. Blasko, G., Feiner, S., and Coriand, F. Exploring 12. Willis, K.D.D. and Poupyrev, I. MotionBeam:
interaction with a simulated wrist-worn projection designing for movement with handheld projectors.
display. Proceedings of the Ninth IEEE Proceedings of the 28th of the international
International Symposium on Wearable Computers, conference extended abstracts on Human factors in
(2005), 2-9. computing systems, (2010), 3253-3258.
2. Cao, X., Forlines, C., and Balakrishnan, R. Multi- 13. Yilmaz, A., Javed, O., and Shah, M. Object
user interaction using handheld projectors. tracking. ACM Computing Surveys 38, 4 (2006),
Proceedings of the 20th annual ACM symposium on 13-es.
User interface software and technology, ACM
(2007), 43–52. 14. Yoshida, T., Hirobe, Y., Nii, H., Kawakami, N.,
and Tachi, S. Twinkle: Interacting with physical
3. Cao, X., Forlines, C., and Balakrishnan, R. Multi- surfaces using handheld projector. 2010 IEEE
user interaction using handheld projectors. Virtual Reality Conference (VR), (2010), 87-90.
Proceedings of the 20th annual ACM symposium on