Google’s latest research aims to make mixed reality videos a little less alien

Consumer VR has been around for a nearly a year and there are a ton of questions floating around that still boggle even the sharper minds in the industry. One that’s been answered (to some degree), however, is how on earth developers can show consumers what VR is actually like without needing them to experience it in person.

Researchers have made pretty quick work of answering this question by their work in mixed reality setups. Through the use of a green screen and a lot of technical trickery, the process places a headset-adorned VR user in the virtual environment they inhabit.

You can see just how awesome the whole capture process is by checking out this video of Conan O’Brien playing some VR at YouTube’s VR mixed reality studio.

Game publishers like Owlchemy Labs and Radial Games have done some of the most extensive work on mixed reality setups and have shared their findings with the greater VR game dev community. Many of the issues of reckoning a human avatar in a digital world have been accounted for in these efforts but today, Google revealed in a blog post that its been working on a strange little project to go the last mile in making these MR videos even more realistic by bringing the user’s face back into these videos.


Through the work of a facial capture program, eye-tracking and a little computer vision trickery, Google’s Machine Perception team has devised a system for getting realistic facial responses to seemingly “shine through” a modified VR headset, complete with with eye movements that logically follow the participant’s actions.

This “headset removal” solution obviously wasn’t some sort of massive issue plaguing VR content creators/sharers but it’s cool to see Google going to thoughtful lengths to solve problems that most consumers didn’t even see as a problem. Something as simple as bringing a user’s eyes back into VR makes the tech appear a little less alien to a consumer observer.