Microsoft recently invited me to join in a one-on-one chat with Alex Kipman, the pioneer of mixed reality, the technical collaborator for Windows mixed reality. The difference was that it was not on Zoom or Teams to gather around a virtual table which shot up in my home office.
A holographic cartoon version of Kipman hovered in my space, and I walked around it. The only thing I needed to connect was a visor I was wearing over my face: the Microsoft HoloLens 2. My test drive of the HoloLens 2 at home showed me for the first time ever where the AR- glasses are probably on their way. And also the challenges that still need to be solved. Microsoft Mesh, a technology that promises a way to radiate people into the same shared virtual space, shows incredible promise. But the hardware that will make the most of it has not quite arrived.
Look at this:
Meeting with Microsoft’s Alex Kipman in the HoloLens …
2:18
The HoloLens 2 has been around for over a year, but not for you or me. It’s sold as a business device, meaning it’s a $ 3,500 headset designed for people in the workplace who can afford it. Unlike VR headsets, they are not really designed to play games. And Microsoft never sent out review units of the HoloLens 2: my demonstrations were always in controlled spaces, for limited amounts of time. When Microsoft offered to send out a lender HoloLens 2 as part of the mixed reality software announcement, I was very intrigued. This is still a very new device for me.
Remember it is an AR headset, not a VR headset: the lenses are transparent. The HoloLens 2 covers glowing virtual objects that seem to exist in the real world. The only other headset like this is the Magic Leap One, also a business device (which I could try once a week or so in my office). It’s not about entering a virtual space, but being in my own space and sitting well on it. All the Marvel and Kingsman and Star Wars dreams about holograms you can communicate with, well, that’s Microsoft’s goal. Since Qualcomm and Facebook and maybe Apple (and others) are working on AR headsets, the HoloLens 2 looks like the prototype for the next one.
The HoloLens 2 is not quite up to the mark, but no one is. Yet it can now come closer than anything else.
HoloLens 2, Oculus Quest 2: both standalone, both easy to start. One’s AR, one’s VR.
Scott Stein / CNET
It reminds me, oddly, of the Oculus Quest 2
The headset is surprisingly compact and about the same size as (although it feels lighter than) the Oculus Quest 2, Facebook’s standalone VR headset. While the Quest 2 is $ 300 and the HoloLens 2 is over $ 3000, there is a spiritual similarity with both. They are both standalone devices that do not require computers or telephones to use. They both fit easily over my head and fit over my prescription glasses.
The independent and easy-to-use feel of both serves a similar purpose: get people fast in VR (or AR) and without cables and odd interfaces.
This is where the similarities end.
Look, no controllers
The Oculus Quest and HoloLens 2 both allow manual tracking, but Facebook uses it as an alternative to the Quest controllers. Manual tracking on the Oculus Quest works surprisingly well, but the HoloLens 2 has no controllers at all: everything is done with your hands. This is where the HoloLens 2 shines … and also has awkward moments.
To touch virtual things, such as buttons or keyboards, I stick my fingers out and tap it. To grab an object, I pinch the edge. I open the HoloLens menu by looking at my wrist and tapping glowing on a button that appears there. To control things from afar, I open my hand and throw a beam like I’m Vision. There is a sense of supernatural forces flowing through the HoloLens interface.
On my own I try to play a game called Roboraid on the HoloLens 2, where things appear from my walls – I tried a variation of this game on an E3 demo many years ago, but at home I use my hands to play. This is a lot that requires HoloLens 2 to pinch and tap my fingers together. The arm gestures can become tiring. I want simple shortcuts. And also, a controller would be nice. I can get no feedback like vibration, this is where a wristband or ring or neural input technology comes in along the way, as Facebook planned. A kind of controller can help make gestures more minimal and even make me feel what I am doing.
Even with the limits of the smaller than the desired display of the HoloLens, I can draw 3D in my room, scratch lines from my bookshelf and draw the actual objects. I put virtual objects together with rights. The virtual, glowing people stay in their place, and when I come back later, HoloLens 2 on, they’re still there.
The flip-down lenses of the HoloLens 2. The headset can also keep an eye on my eyes.
Scott Stein / CNET
Eye detection: a technology waiting in the wings
The HoloLens 2 also has eye tracking, something that current VR headsets are not for business. Eye detection is subtle, but it allows me to look at an object – like an open app window across the room – and say ‘app closed’, and I know which one to lock. For moments where I talk to people in AR, they can see my virtual avatar eyes moving because the eye detection notices where I am looking.
Facebook CEO Mark Zuckerberg wants eye tracking on future VR and AR headers for the same purpose of mapping facial expressions and eye movements to realistic avatars. However, Microsoft’s technology is business-oriented and includes the use of eye-tracking data in specific, secure cases. How will and be used and divided into chapters and glasses in the mainstream? We do not know yet.
A floating virtual desktop full of windows
I demonstrated Microsoft’s software and also tried some other applications. I sit down and try to open web browsers and then play a game or two (yes, there are some). What really struck me was how windows could appear on my desk and in any shape I wanted. I could get up and they would stay stuck there. They would even be there the next day.
Qualcomm’s AR glasses are designed to connect connected screens for phones and computers. What I see on the HoloLens 2 feels like a preview of the glasses and what they will eventually be able to do.
On the HoloLens 2, I’m limited to using my hands (though I can connect a keyboard). I would love to see what it is like for my laptop to suddenly have extra windows and monitors pop up in the air when I put on my future smart glasses.
The cameras on the HoloLens 2, which can scan my room deeply like the member 12 of the iPhone 12 Pro.
Scott Stein / CNET
The exhibitions are not yet perfect
The limited field of view of the HoloLens 2 feels like a large window floating in front of me where glowing 3D things appear. But the window is not wide enough, which means I have to move my head to take things into the room that I do not know.
The screen also has a slightly hazy rainbow-like quality. This is not the perfect live show I would expect on a monitor or even recent VR headsets. If I want to use an AR headset to watch movies or play games, I want to develop something more. It’s not easy with a transparent lens, but maybe Micro LED technology can help improve things soon.
What will the killer programs be?
Microsoft’s HoloLens 2 uses communication and telepresence as its killer programs for business. It can also be excellent due to instructions in the workplace. But what would be the killer programs for AR glasses sold to everyday people? Would it be fitness? Games? Virtual movie glasses? An extra monitor that can go anywhere?
No one has figured it out yet. Companies like Niantic, makers of AR’s ultimate killer app, Pokémon Go, investigate play on AR glasses with a HoloLens 2. Microsoft’s headset is not meant to go anywhere. It is not wonderful in bright daylight; it looks large and helmety; and the battery life is not long. But this is probably the best prototype I have ever tried for what AR glasses should do next.