How To Use Our XR Keyboard For Unity¶

Keystrokes: Down–Push or Up–Release?¶

Initially when building these keys we activated the keystroke on the down–push. This gave the most responsive feel. Users didn’t have to be too accurate about their hand movement to activate the keys – as long as you press down on the right key it works.

Further down the line we realised we needed to handle long presses of keys (more on this later), so we moved to activating keystrokes when the key was released.

We expected this to be somewhat detrimental to the performance of the keyboard because you would need to be more careful about how you activate the key through press and release. But we were wrong. As long as you press down the key you intend, the Interaction Button allows for some movement of your finger after you press while still triggering a successful up event when you pull your finger away.

There are definitely faster ways of typing with your hands in VR, but it is unlikely that any of them will be as instantly usable as a traditional push–retract button.

Key activation on down vs up

Event Feedback¶

This style of button is great for a sense of familiarity, but the lack of haptic feedback limits the reliability of the interaction – you can’t feel the sensation of the key activating in the way you do with a physical keyboard.

Lack of physical feedback when using hand tracking is currently one of XR’s biggest regressions from reality and a problem we are trying to remedy at Ultraleap. However, as contactless haptic systems aren’t widespread yet, this example relies on audio–visual feedback.

Our keys use both colour change and sounds to convey touch. This is one of the reasons activating on up–release works so well because we still use an audio cue to let you know when the event takes place.

The Number Row¶

Numeric characters are a very common feature of usernames and passwords, which are likely use cases for this type of keyboard.

The QWERTY layout only really helps us with where to place the characters of the alphabet. When it comes to number keys there are several potential solutions to choose from.

Traditional physical keyboards often feature a horizontal row of numbers across the top, but many also offer a number pad to the side. We explored both options to see which would perform best in VR.

We felt that the number pad to the side of the keyboard might offer some benefits in terms of reliability and speed of entry, just like it does on a physical keyboard. But in practice it felt cumbersome to use and was prone to error. This was down to the increased width of the keyboard – you had to turn your head before reaching out with your hand. On the face of it, this doesn’t sound too bad, but in practice turning felt awful!

Typing on a keyboard is a high–precision task that requires a high level of situational awareness so turning is a problem. It causes a huge context switch and ruins flow.

Once you’ve turned, you need to first reorient yourself, and then search through the keys on the number pad to find the key you were looking for as you recalibrate your movement to fit the new arrangement. This is slow and increases errors.

Even a fun animated pull out effect couldn’t save the number pad from feeling too wide

So, we needed a new solution that doesn’t need you to turn, and doesn’t cause a context switch.

Enter…the number row!

Located above the existing QWERTY panel, this sits within your field of view when looking at the centre of the keyboard. It’s easy to reach with a small hand movement. Placing it here immediately felt natural and keeps you in your flow – no turning required.

We did find that the addition of another row of keys felt somewhat overwhelming. A lot of keys all on the same panel resulted in a layout that felt crowded and hard to focus on. To solve this, we detached the number row from the keyboard so that it floats just above the first row.

This subtle design change had a big impact. It created a sense of space, kept the row distinct from the alphabetical characters, and made it easier to take in all the information at a glance.

When placed close to the alphabetical characters the numbers are hard to differentiate, so it feels cluttered	When spaced apart it feels less cluttered and it’s easier to distinguish the numbers from the alphabetical characters

The number row in action

Accessing the Accents Panel¶

We wanted to design for the untrained user every step of the way. This didn’t make life easy for us, but time and again we kept running into the same solutions:

1. Start from a place of familiarity

2. Ensure that the action doesn’t create a context shift and detracts as little as possible from the user’s state of flow

A particular challenge was to make this keyboard usable for everyone, not just English speakers. We needed a way of adding characters with diacritics (e.g. letters with accents or umlauts above them). By applying these principles we created an easy–access accent panel.

We identified 34 extra characters we wanted to add – definitely too many to simply add on an extra row and hope for the best! So we followed our now tried–and–tested process:

Start from a place of familiarity

Physical keyboards offer poor implementations of diacritics, often relying on hidden keycodes which can be complex and hard to memorise (e.g. alt+0232). But smartphone implementation is much more elegant – long pressing a key to activate an accent panel.

We had initially wanted to avoid smartphone keyboard design, but it seemed the most familiar design that wasn’t a physical keyboard.

Immediately after implementing long press, it felt completely familiar. Like a natural part of our keyboard.

But we were still left with where to place our accent panel after the long–press… enter step two:

Ensure that the action doesn’t create a context shift and detracts as little as possible from the user’s state of flow

We had a few potential solutions for placing the accent row:

• Temporarily replacing the number row

• Floating above the centre of the keyboard

• Floating near the long–pressed key

Of these prototypes, the last fulfilled our criteria the best.

By placing the accent panel near to the long–pressed key you are already looking in the right location, and your fingers are already close enough to reach out and press the key you want. This feels effortless and avoids context switching.