I
n 1895, audiences sat down to watch “L’Arrivée d’un Train en Gare de La Ciotat,” an early film that showed a train pulling into a station. Legend has it that when viewers saw the train barreling toward them they panicked, because they hadn’t experienced the new medium.
Whether the story is true or not, virtual reality filmmakers are recalling it with interest. Less than a year after the release of the first modern VR headsets to the public, filmmakers are in the same position as the film industry was in the late 1900s. The technology is new, as are viewers’ expectations. No one knows what type of content will work best, or even what type of content will be possible a few years down the road.
“In virtual reality we’re right at that beginning stage again,” said Mark Walsh, a Pixar alumnus who leads an interactive VR studio called Motional Entertainment. “It’s not that we need a different kind of film stock or a different kind of camera. We just need more artists to be playing with the medium.”
So filmmakers are borrowing techniques from movies, video games, theater and oral storytelling and seeing what happens. The early results are promising. VR has the potential to wow audiences with intimate access to people and places once contained to a two-dimensional screen, and that’s drawing veterans of Hollywood, animation studios and the gaming industry. Together, they are creating a new reality.
Beware the train
In 1895, audiences panicked when watching “L’Arrivée d’un Train en Gare de La Ciotat,” thinking the train was coming right at them.
Building intimacy
It’s unclear if VR films should even be called films. On one end of the spectrum, there are traditional movies; you sit in a chair and watch something happening in front of you. On the other end of the spectrum are games that require a high degree of participation.
VR films fall somewhere in the middle and, depending on which one you are experiencing, the expectations for you as the viewer can vary wildly.
“Invasion!” is an early standout film that makes use of one of VR’s most interesting mechanisms: eye contact. The film, which was created by the 15-person team at Baobab Studios, places you in the body of a rabbit.
You don’t need interaction to build empathy.
On a vast frozen lake, you encounter another rabbit who looks straight into your eyes and smiles sweetly. Instead of the jarring feeling of a character breaking the fourth wall, the gesture places you within the story as a central character.
This is the transportative nature of VR. While it’s exhilarating to hack a zombie to pieces with a virtual axe, it’s also powerfully immersive to simply be near a character we care about. Like every medium ever, storytelling is what we humans crave. And now we can take an intimate seat within every story.
The Future of VR Storytelling | 5:27
// VR filmmakers face a bit of an identity crisis here. When people watch a movie, chances are they want to relax. They don’t want intense participation; if they did, they would play a video game instead. You don’t need interaction to build empathy.
Entering a story adds another interesting truth: Once a viewer is a part of the action, they gain a feeling of responsibility.
“There’s this constant struggle between ego and empathizing,” Baobab Studios CEO Maureen Fan said. “Your mind is thinking about yourself and your own ego and the other characters. You may not be thinking about those characters as much as if you weren’t a character in the world.”
In Oculus Story Studio’s “Henry,” a lonely porcupine throws a birthday party for himself. The studio realized early on that Henry can’t make eye contact with the viewer, because that would mean he’s not alone. Instead, the viewer plays observer to Henry’s painful solitude. The tension of being so close but so far is heart-wrenching.
So as novel as it is right now to place the viewer in the story, it doesn’t always make sense. If a viewer is supposed to be building understanding of a Syrian refugee, it can be distracting to make them wonder how they should help within the movie.
A new relationship with viewers
Once you step into a virtual world, it’s natural to want to reach out and touch a character. But most films don’t let you do that. A viewer might not even know if they’re entering a game or a film. In these early days, directors are learning the importance of cueing the viewer on expectations.
“You don’t want to dangle a delicious piece of cake in front of somebody and then not let them eat it,” Walsh said. “If you’re going to put something within their reach, they expect to be able to reach it.”
When a viewer pulls on a VR headset, they gain control of the camera. They become the director. If a filmmaker forgets that, they risk losing the viewer’s attention at the most critical moments. Or they risk overshooting the limits of human attention.
“We can only keep track of two-and-a-half things at a time,” Oculus Story Studio creative director Saschka Unseld said. “Once you can not only look around but you can move around in an experience, that takes over two of those attention things you can keep track of. You have about half left over for a voiceover.”
Basically, don’t make it too complicated. Stick to a story that flows with viewers’ innate expectations. Filmmakers can regain control of their films with suggestions and optical cues. Audio is one of the strongest options; make a noise, and people are sure to look for its source.
In “Invasion!” Baobab’s animators used natural frames like mountain peaks to guide the viewer’s eyes. They also created natural stopping points on the frozen lake’s expansive surface by making certain areas lighter or darker.
More obviously, the viewer can follow the bunny’s eyes. Her ears prick up with curiosity or she cowers low in fear. It draws the viewer’s eyes toward the arrival of an alien ship, which flies into view and then dips behind a patch of trees.
The film’s team found that after a few seconds, viewers start to look around, wondering if they are missing the action. And that means they might not be looking in exactly the right place when the alien ship reemerges. The film forgoes long, drawn-out periods of suspense in favor of keeping the viewer clued in, according to CCO Eric Darnell.
It’s okay to be blunt, too. At the Game Developers Conference in March, I sat down on a virtual beach and met the characters of Motional’s “Gary the Gull.” I couldn’t move my body or touch Gary, but he asked me questions. Then he looked at me expectantly until I realized he wanted me to nod my head yes or no. After a stilted but functional conversation, he tried to steal from my cooler.
But it all comes back to characters. Without curiosity and delight, viewers lose interest and start looking around for something else to do.
Motional’s Mark Walsh, who led animation on Pixar films like Ratatouille and Finding Nemo, designed Gary. While the animation in “Gary the Gull” is somewhat crude, the characters are irresistible. Gary talks in a charming, over-the-top voice that you can’t help but engage with.
“The big trick is creating a personality that people will identify with or be entertained by — that has flaws that make it lovable or likable,” Walsh said. “In virtual reality, we’re meeting these characters. So the way that we approach creating empathy or creating entertainment with a character is for me completely different.”
Gary’s big trait is that he’s a con man. In a film, creating Gary would entail animating a gull conning another character. But in VR, he has to prove it. He has to con you as the viewer to build out his character. It’s a more difficult sell, but it results in a more authentic connection with the character.
Traversing the uncanny valley
VR tricks our brains into believing that what we are seeing is real, which creates some interesting nuances, according to former Jaunt VR CEO Jens Christensen. Actors have to act less like actors and more like real people or else it’s distracting. At GDC, some VR developers argued VR experiences shouldn’t have music at all, opting instead for realistic ambient noises.
One of the most difficult challenges is overcoming the technology itself. Giving viewers the ability to determine the shot means each frame must be rendered almost instantly if a filmmaker is working with animation. VR only looks good at an incredibly high definition, so getting the desired look requires some trade-offs. For example, very few VR films allow the viewer to walk around because it would make it so much more difficult to render.
In “Henry,” the filmmakers decided to apply texture in only the most important parts of the film. The team spent a lot of time making Henry’s quills and fur look just right, but since the viewer only sees one side of him, his back is just empty space. He also has a simplified shape and limbs. Many of the features of his home are hidden in shadow or filled with less detail, because they just don’t matter as much.
Cutting and panning can be problematic in VR because, done wrong, they make viewers queasy.
The obvious challenge of shooting a live-action film for virtual reality is hiding any evidence of the shoot. The director and crew can’t be in the shot, and neither can boom mics or any other type of equipment. The VR film sets I’ve attended took place in small rooms the crew could quickly exit, or involved dashing away to take cover after each scene reset. It’s something that forces even actors to relearn their jobs.
“You have to remember in VR that no matter where you are, you’re in the shot,” Christensen said. “You have to remain in character for much longer than you’re used to.”
Lighting and other specialty equipment must be subtly integrated into the set. Directors seek out natural light or build out existing sources like streetlights or campfires.
Traditional film techniques have to be rethought as well. Cutting and panning can be problematic in VR because, done wrong, they make viewers queasy. Christensen still believes they have their place, but a lot of filmmakers are staying away from them altogether for now.
New tools emerge
A few years ago, shooting a virtual reality film involved assembling GoPros or DSLR cameras into a ball and then laboriously stitching all the footage together into a spherical film.
Now, there are options. Consumer cameras are hitting the market, but there are also professional-grade cameras like the system built by Jaunt. Editing platforms that ease working with 360 degrees and interactive points are also hitting the market.
Animators usually opt for game developer software, which is better suited to interactivity and a work pipeline that lets multiple teams work on a film at once. But there are also emerging options that let animators work directly within virtual reality. Oculus recently unveiled Quill, which provides a 3D space in which to paint. Unseld described it as a closer representation of an artist’s intentions; it’s not photoreal, but it’s real brushstrokes — a style that works especially well for the right types of stories.
Filmmaking could be the world’s entry point to VR.
One of the most interesting technologies on the horizon is stereoscopic VR, which creates a limited depth of field that mimics the way the human eye sees. This tends to feel even more real than the most common types of VR film, which have everything in view in focus.
Lytro is working on a camera in this style, and NextVR has demoed stereoscopic films. The technology could mimic our eyes even more with the help of VR headsets that track gaze, allowing the film to refocus based on where you’re actually looking.
Filmmakers are also hoping to see lighter headsets become available in the next few years. Films currently tend to max out at around 10 minutes. That’s partially because it’s so labor-intensive to produce a VR film, but also because that’s all viewers can stand.
“It can be uncomfortable to watch a single piece of content for much longer than 10 minutes unless it’s very good,” said Neil Graham, who leads VR production for Europe’s Sky channel. “But this will increase as headsets get lighter, and as stitching and motion improve, which it is doing by the week.
//
Here’s to experimenting
In these early days of VR, it’s tempting to prey on audiences’ emotions, just as “L’Arrivée d’un Train en Gare de La Ciotat” supposedly did. Attend any VR gathering and you can expect to be terrorized by zombies and shark attacks. The emotions you feel in VR are real and raw. There’s no longer a screen dividing you from the virtual world. You’re there.
But the medium has already drawn storytelling masters who want to move VR beyond viewers’ screams and sweaty palms. There’s a healthy pipeline of new talent on the way, too. They’re applying decades of lessons from film and gaming to create an entirely new mashup. While some of us are gamers, most of us are cinema goers. And that means filmmaking could be the world’s entry point to VR.
“With everything we keep doing, we’re starting to build vocabulary and ideas to in a year or two create something that is so unlike movies that you won’t even compare it with that anymore,” Unseld said.