Spatial Design for 2D Screens¶

A window to a 3D world - spatial input with a non-spatial display¶

The main focus for our RAD team in the last few years has been creating XR experiences (virtual, augmented, and mixed reality). Designing for hands in a monitor-viewed 3D world shares some similarities with designing for XR - both of the mediums are, in a way, spatial design. Your hands have a presence in a 3D space, like in XR. But in this case, your monitor becomes a “window” to a 3D scene.

Your monitor acts as a “window” to virtual reality.

The differences between the two mediums become clear in practice almost immediately. While we were aware of some of the challenges that tabletop tracking (the camera placed on a table) presented ahead of our game jam, we discovered a lot of interesting things to tackle and gained a lot of insight into the challenges that the setup encounters.

Key challenges when designing for non-spatial displays¶

When designing a 3D experience, there are two key differences between 2D displays and XR headsets that affect the design:

• The lack of depth perception. No headset means no sweet stereo rendering for your eyes (except if you’re using 3D displays - to be explored!). It is now your responsibility to build depth in your scene when your users lack all the most common 3D cues from the environment.

• Both your real-world hands and digital hands exist simultaneously in different realities. In XR, your digital hand presence and physical hand presence are in the same place and blend into each other. When viewing a 3D world through displays, you are basically controlling a copy of your real-world hands. Your “puppeteered” virtual hands can occlude your objects, and unlike in XR and real life, moving your head won’t solve the problem, as the camera is in a fixed position.

These two key differences cascade into a lot of tricky design challenges that you won’t have in VR or AR. Here are some of the challenges we encountered:

The next chapters will tell you how we approached some of these challenges.

Exploration time! Prototypes, prototypes, and even more prototypes.¶

Depth - Where are my hands?¶

To interact with objects with hands, you need to know where your hands are relative to the objects in the 3D world. Traditionally our depth perception comes from a combination of a wide variety of stereo and monocular cues.

When using 2D monitors, some of the most important depth cues that are missing are:

• Stereo separation of our eyes (stereopsis) - The difference between the “image” we receive from our left and right eye.

• Motion parallax - The difference in the movement speed between near and far objects when your head is moving around.

Other depth cues lacking in 2D displays include eye focus (accommodation) and convergence.

Luckily, there are still some depth cues left we can use!

---

Shadows and reflections¶

Shadows everywhere! Shadows proved to be one of the most impactful ways to know where your hands are floating in the scene. Shadows both “ground” your hands to the environment and gives a cue to how close they are to the surface catching the shadow. One technique we also saw in other experiences was replacing shadows with reflections, for example from a water surface. We didn’t end up using a reflection in our own experiments as it didn’t fit with our environment, but if you are looking for an alternative to shadows this would be a good option to explore.

Ground receiving shadows

Finger casts shadows towards the object you are interacting with.

Reflection on the water. Video from V2 Playground experience.

Tips for shadows:

• The direction of the light forming the shadow matters! We noticed it is way easier to understand the position of the hand if the shadows are projected directly underneath it.

• Fingers form shadows on the object they are approaching. You can use this to provide cues on how close the fingers are to object surfaces. In our case, we ended up adding some extra real-time lights to create the surface shadows - as the experience is run on a desktop rather than mobile, performance limitations are less of an issue so we can have multiple real-time lights in the scene.

---

Occlusion & reference objects¶

You can use occlusion to give cues on where the hands, or objects, are related to the other objects in the scene. We ended up experimenting with the following extra occlusion objects:

Occlusion helpers

You can add extra objects for occlusion, such as lines.

Chaining reference objects

UI buttons are connected to the ground geometry with lines to understand where they are in depth relative to the central object.

Just any objects, really.

Having any objects in the scene help with realizing where your hands are relative to the world, as the hands start occluding with the geometry.

Tips for occlusion and reference objects:

• Be careful if using translucent hands. We noticed that not only do objects occluding hands matter but also hands occluding the objects. The perception of depth drops if you introduce hand transparency.

---

Depth helper effects¶

We can use the fact that we aren’t bound by the real world to our advantage and add some special depth helpers! Here are some that we played with:

Proximity-based cursors on your hands!

Hooking things like an additional light or a cursor to your finger allows the surface to react to the proximity of the hand.

Floor grid highlight

Reference surfaces can have reactive colors based on the hand’s position.

Helper lines

Extra helpers to explain the distance from hand to the surfaces.

Attachment “leash”

A leash that hooks up to the closest object you can grab. In this case we experimented calculating the closest object in screen-space.

Tips for effects:

• Free your imagination! Real-world limitations won’t affect you in virtual worlds. Test out effects that might fit your own use case.

• Combining screenspace-calculated effects with 3D may cause confusion. With the leash effect, we tried out calculating it in 2D space instead of in 3D - the attachment leash connected to the closest object that your hand was in the screen, not in the 3D space. While our users could generally select things quite fast, it was causing confusion in the depth of the objects.

Enter Dog Racer!¶

At the end of our jam, we ended up combining a bunch of our interaction experiments into a small Bamzooki-inspired game named “Dog Racer”.

In Dog Racer, we applied a number of the learnings from our interaction prototypes. We specifically included the following:

• Shadows: We are casting shadows both to the ground and towards interactable object while painting.

• Depth effects and helpers: Floor grid highlights, proximity-based cursor effects, attachment leash (without the visual line), reference object chains.

• World space buttons: Using world space buttons to navigate the application.

The experience has three main interactive parts:

The result here leaves a lot to be improved outside the time constraints of the game jam, for example, making the racing part more interactive and polishing some of the components and 3D assets. However, we saw that people were having already fun with this prototype - and we certainly learned a lot while making it! 😊

You can download and try out the experience here:

• Windows

• MacOS

Conclusion¶

Designing for non-spatial displays is no mean feat, but hopefully this blog has given you some inspiration on making your own 3D experiences on 2D screens. We’ve open sourced some of the main experiments discussed in this blog - feel free to download & test them out! You can find them in this repository.

The experiment repository has the following interactions available:

Finger painting of the dog was done with mixandjam/Splatoon-Ink: Recreating the Ink System from Splatoon. - .

We’re looking forward to see what you create with Leap Motion Controller 2!