Facebook CEO Mark Zuckerberg wants to track eye detection and facial expression in future iterations of Oculus Quest.
In a wide interview with The information, Zuckerberg spoke about the company’s strategy and direction in the AR / VR space.
He said Facebook is already working on its next VR headsets:
‘Because of how hardware is developed, you need to know what your next three products will look like at the same time. It’s not like software where we change it every few weeks. We now have product teams working on the next few generations of virtual reality and what Quest 3 and 4 will look like.‘
Zuckerberg unveiled one of his most popular personal features in the upcoming Oculus headset; eye and face detection to drive avatars into social experiences:
‘One of the things I’m really excited about for future releases is getting introspection and facial recognition., because if you’m really excited about social presence, you want to make sure the device has all the sensors to really animate realistic avatars so you can communicate well.‘
Later in the interview, he talked about some issues to make it happen:
‘On the VR side, I think the biggest things we are focusing on a lot right now: how can you pack more sensors into the device to create a better social experience? To make each sensor work, you need more CPU power, which generates more heat and causes all these different problems.
When I think about where you are with VR today, there are some great games and different experiences, but I would like to reach the point where you have realistic avatars of yourself, and where you can make sincere eye contact with someone and get real expressions that are reflected on your avatar.
So, what do you need for that? Well, you need to be able to do eye tracking to make eye contact. You need to have some facial recognition or realize what is going on with the person’s expressions to let the emotions come through naturally.‘
VR headsets today with eye tracking or face tracking are aimed at business buyers. Pico’s Neo 2 Eye is the first standalone eye-catching headphone, priced at $ 900 with a focus on enterprise use cases. HP’s PC-tethered Reverb G2 Omnicept Edition has eye tracking and face tracking, but is also focused on business use cases with prices yet to be announced.
Facebook see avatar research
In March 2019, Facebook first showed its long-term investigation into photoreal ‘Codec Avatars’.
Powered by machine learning, the avatars are generated using a specialized recording camera with 132 cameras. Once generated, it can be animated in real time by a prototype VR headset with eye tracking and facial tracking cameras.
At the first launch of codec avatars, Facebook warned that the technology was still ‘years away’ for consumer products. Sending this kind of photorealer avatar requires a number of breakthroughs.
Facebook’s Avatars Today
Oculus Quest today has a built-in basic avatar system, Oculus Avatars. It is used in some programs as Poker Stars VR and Stam XR DJ, but not much else. Platforms like Bigscreen, VRChat and Rec Room use their own separate avatar systems.
Current headphones do not have eye or face tracking, but Oculus Avatars uses a neural network to simulate lip movement, and developers can set priority-setting gaze targets to simulate eye movement.
Back in September, Facebook announced that the new Facebook Avatars system would replace Oculus Avatars. This is apparently an evolution of the current Facebook VR avatars used in Facebook Horizon and the beta for the new venues. Staff working on the project include former Pixar animators.
Facebook Avatars apparently takes a step backwards in artistic realism, but adds a full torso and simulated arms.
In today’s interview, Zuckerberg announced that Facebook will be sending Avatars this year, saying that over time it will become more realistic, indicating that the company intends to take a recurring approach. Zuckerberg also commented on Facebook’s cheap hardware strategy.