HTC Vive Pro Eye: Why the company added eye-tracking to VR

At CES 2019 this week, HTC continued to push the high end of virtual reality forward.

HTC added eye-tracking to its new Vive Pro Eye high-end virtual reality headset. This adds new features like foveated rendering, advanced analytics for developers, and eye-tracking input for game and application developers.

HTC showed a few enterprise and game-related features that take advantage of the eye-tracking technology. One example was Lockheed Martin’s cockpit training application for fighter pilots. Another was the use of eye input for the menu of a MLB baseball game demo, and other was the

Jeremy Horwitz covered the press event at CES 2019, and he wrote his impressions of the HTC Vive Pro Eye. I caught up with Rikard Steiber, the president of Viveport and senior vice president of virtual reality at HTC, in an interview at the Wynn Hotel in Las Vegas. The new Vive is expected to be released in the second quarter of 2019.

Here’s an edited transcript of our interview.

Rikard Steiber: At this CES, the main thing is to extend our leadership, not just around hardware, but also around content. We want to bring that together with a great software experience around Vive Reality.

Eye tracking is a natural evolution. A lot of customers we talk to, especially in the enterprise space, want to use this for training applications. There are three things that are important here. One is interaction. Instead of pointing and clicking, you just naturally look. The second is the social interaction. As we have more and more avatar interactions, having actual eye contact is a great thing. The third is foveated rendering, especially when we move to more streaming. You get very high clarity where you’re watching, but you can save on the data where you’re not watching. Those are the main things we’re trying to showcase.

In the beginning you look at things because you know you have to look at them. Eventually you’re just going to interact with everything in a more natural fashion.

GamesBeat: Does almost everything have foveated rendering here?

Steiber: No. The developer needs to make sure they leverage the technology. It’s the same with eye tracking. I think we’ll see a lot of developers upgrade. They’ll change from pointing and clicking to looking and doing something. A lot of the demos here were done in just a couple of weeks, the upgrade. There’s a fairly easy process for developers.

You probably heard about Vive Cosmos. We’re not telling everything. I think the main thing is that we want to make sure VR becomes much more accessible to everyone. One thing is that we want to make sure it becomes much more user-friendly, both when it comes to the hardware and the setup. It’s inside-out tracking with no base stations. The design is much lighter. It’s easy to get into, and you can flip it up, so it’s optimized for a broader audience. Still premium, very high quality.

The way you’ll get into the hardware, we’ll have Viveport Infinity that we announced, which is essentially Netflix for VR. You can access hundreds of titles as much as you want for a low monthly fee. We’re going to push that. When you’re in VR, you don’t have to think about buying anything. You can just use what you want.

Then you have the Vive Reality system, which is a new interaction platform. You land in our home space, where you have all of your stuff. You can invite your friends there. From that, you have easy access to your store, your content, your video player, your browser, and your social experiences. When you’re in Population One, you can open something called the Lens. You press the button and this portal opens up, which allows you to move very easily from Population One to, say, your Mozilla browser and check something out. Or maybe you have a meeting, so you can check out the meeting app we have here. If someone’s pinging you and you want to have a social interaction, it’s very easy to transport yourself between these different worlds.

The idea is that we want to get the hardware, the content, and the software platform working in a way that’s much more integrated together. We’re targeting people who are interested in VR, but who maybe think it’s too much hassle right now. Those are the people we want to bring on board.

GamesBeat: You don’t have a date for the Cosmos yet, right?

Steiber: It’s a first look today. In the next couple of months we’ll have more details around launch and pricing and tech specs and so on.

GamesBeat: What else are you saying about it for certain? It’s not wired?

Steiber: It’s tethered to your PC. However, what we announced yesterday is that when you want to bring this thing with you, you’ll be able to connect it to other computing devices. You can have dual mode. It’ll also be much more extensible. You can see these four cameras. There will be all kinds of extensions coming to this thing.

GamesBeat: Can you use it standalone, without connecting to something?

Steiber: No. It needs either a PC or a mobile device, a computing unit.

GamesBeat: So it would be tethered with a wire down to your pocket, something like that.

Steiber: We’ll give you more information soon.

GamesBeat: Where does this fit in the spectrum, then, of what you’ll have out there?

Steiber: We want to be the leader in the premium VR segment when it comes to the PC, as well as stand-alone. We think that having something that can bridge those two worlds—we think that’s a segment and it’s going to be important. This is much more accessible. You don’t need to set up external base stations. You just put it on and you’re there. If you want to take it with you there’s a model for that as well.

We’re not going for hardcore gamers, but a broader audience, people who are curious about it today. We’ve done a lot of research trying to figure out what it would take for people to move into VR. If you have something that doesn’t necessarily need a high-powered gaming PC — if you can use a lighter computing device, all the way down to something like a mobile phone, but things in between as well – it’s much more accessible to a broader audience. If it’s portable, it doesn’t need to be in one room. You can move it into the living room or bring it elsewhere.

GamesBeat: Eye tracking sounds like a high-end feature. Is that true now? You put that in the premium hardware first.

Steiber: You’ll see it if you try the Major League Baseball. It’s a big difference for an entertainment experience and the social interaction experience. But I think you’re right. It’ll start with enterprise, because they’re willing to pay that extra premium. Over time, though, I think that will be a standard feature of how we interact.

GamesBeat: Foveated rendering, does that help when you don’t need as much hardware to run an app?

Steiber: What’s important is that, if you have video, 360 video, all those kinds of experience, it’s getting much better. You don’t want to download these things. You want to stream them in an efficient manner. Of course, if you can have a great, crisp experience where you’re looking, you can save on bandwidth. Especially if you have stand-alone. You want to be wireless. This technology is going to be very important when it comes to having that great visual experience.

There is so much streaming entertainment content coming from both the PC and the cloud. Making sure you create a great experience with not just efficient data transfer but also good processing—if you’re efficient, you can have a much more optimized experience.

GamesBeat: As far as the kinds of content here at the show, are you curating it in some way? How are you thinking about that?

Steiber: These experiences here are various showcases for eye tracking, including everything from simulation and training, where we think there’s a lot of potential. The MLB experience is great. For some of the interaction there, the eye tracking is such a game changer.

Over here, we announced our partnership with Mozilla. We believe that consumers need to have access to much more content, and web VR will be very important. Mozilla has created a new version for the Vive. It’ll be our native browser. As part of this partnership we’ve also partnered with Amazon Sumerian. Essentially, all the websites we go to normally, Amazon has created tools for them to create 3D web experiences.

We believe that today VR is primarily entertainment, and a few other things as well, but if we can add utilities to do the things you need to do on a daily or weekly or monthly basis—if you can do that in VR, that will be very helpful. We’re showcasing how you can go in, look at your investments, bring in a partner if you want to, and have a discussion. In this case it’s automated, but maybe in the future it will be a real person.

Looking at Population One, gaming is always going to be there. The Fortnite battle royale model is super exciting, and this year we’re going to see even more of these kinds of high-end triple-A titles. We do think that the Nordic tech, where you have some sort of physical activity—they do fitness machines, but they’re making it more fun. You’re playing a game, but suddenly you’ve done half an hour on the bike.

The presentation tech here is really interesting. It has eye tracking as well, but essentially—if you want to do a keynote, you can practice how to deliver presentations in front of people. You can put in keywords, so if you say “like” a lot it’ll flag that for you. It knows where you’re looking in the room, whether you’re sweeping it or just staring. It’ll score you.

My thing with eye tracking—we get a lot of data. We have BMW here that we’re showcasing. It’s a great experience, because you can interact and change colors and see what it looks like in a more natural way. But afterward, BMW gets data on where people have been looking. What are the most important features? Maybe you didn’t know that the steering wheel is important.

The other thing is that if you’re training someone—my daughter is getting close to taking driving lessons. I know what’s going to happen. She’s going to drive off, she’s going to flunk the test, and they’ll say, “Well, you didn’t look over your shoulder.” She’ll say, “Yes, I did!” And they’ll say, “No, we know you didn’t, because we’re tracking your eyes.” We have a number of these use cases. Brands can get information about their customers. In training you can get information on how people are interacting. It’s very useful.

GamesBeat: How concerned are you about the possibility of having too many versions of the hardware? Do you think you might splinter your effort? Do you still want to limit the number of variations you have?

Steiber: We want to have the best hardware when it comes to premium VR. I do think that for eye tracking, that’s a natural evolution. There’s a clear market for it. We also think that for stand-alone, with the Focus, which we’ve launched for the enterprise market here in the states—we have development kits for six degrees of freedom. In some training applications for stand-alone it’s good enough to just have point and click, but with some other things you need to interact. We’re going to make sure we’re the leaders there as well.

With the Cosmos there might be solutions where we want to offer customers the flexibility of having six-six in the home or office environment, but maybe you want a portable mode as well. I can imagine use cases where you’re stationary at the office, but then you want to bring something to a customer. Things like that. Each headset is targeted at a different consumer with different needs.

In the Vive Reality system, we want to make sure you have a great browser experience, a great video experience. You can have things like a Skype meeting in VR, and you don’t have to pay or download anything. You just become a member and you have access to 500 or 600 titles. You still may have to buy the big new titles, but for the main things you need, you’ll just have access to it.

GamesBeat: As far as streaming a title versus downloading it, how realistic is that? Will you still have to select something for download?

Steiber: For Viveport Infinity, the way it works—in the subscription service we have today, we have just south of 600 applications and games. Today you can pick five, and you can swap as much you want, but you can only have five at once. It’s like the old Netflix, where you’d have the DVDs at home and swap them around.

The difference now is that you have access to all 500 or 600. You still have to download them. That’s not changing yet. You just have access to 100 times more content. I think at some point you’ll be able to stream some of these things. We’re looking at how to stream video, which is easier than having interaction with a game, where you have to worry about both upstream and downstream. That’s a bit further out.