How Unity wants to save the film industry

Thomas McMullan Read more June 21, 2018

In a former railway station, a short walk from Potsdamer Platz in Berlin, Unity – the company behind the game engine of the same name – has spent the last few days playing home to hundreds of developers, hosting sessions on the finer points of animation, machine learning, world building, games, movies and everything in between. Somewhere amongst all of that was a young girl’s face, controlled by a gurning woman.

During the conference keynote, Unity showed off a new tool, dubbed the Facial AR Remote Component. In the demo, a young female character was projected on a screen; her expression blank. On strides the presenter, who begins to grimace into an iPhone X. The girl follows suit, smiling, frowning, moving her lips in time with the human counterpart. It was as if Pixar has made an animoji.

(Above: Unity’s Facial AR Remote Component in action. Credit: Unity)

This is notable because, after the gurning is complete, the performance can then be woven into a scene; tweaked to fit dialogue, cinematography, mise-en-scène, all without the need for large production studio. “Facial animations are very hard to do,” Adam Myhill, Unity’s head of cinematics, tells me later. “The uncanny valley is a problem. We’ve got hundreds of thousands of years of evolution wiring our brain to see faces everywhere. Faces in the moon. Faces in my mash potatoes. When we don’t see faces move appropriately we cry foul.

“So when you can use a phone to get 50% or 60% of the way there instantly, your budget of animation time isn’t spent going from nought to 100 any more. It’s a game changer. For some groups, where the highest fidelity isn’t key, it might even be good enough out of the can.”

While the avatar shown in the demonstration did a good job of capturing the presenter’s larger-than-life expressions, it remains to be seen how well this system works for the nuances of sly smiles and slightly raised eyebrows. For Myhill, however, the facial-tracking mocap doesn’t need to be perfect – it just needs to be close enough so that the animators have something to work with:

“What if, when the actor is reading the lines, they can be recorded like this? You’ve got their facial expressions to a reasonable degree, and when you start doing virtual production design you can move a camera to land on the guy’s smile. It’s next level.”

“If Hollywood doesn’t wake up and get cheap, they’re going to get fossilised”

A big driver behind all of this, inevitably, is cost. If a studio can get a head start on its animations by simply aiming an iPhone X at an actor’s face, then it can skip the labour-intensive process of building a performance from the ground up. According to Isabelle Riva, head of Made with Unity, this could be a major saver for the industry.

“Film is a place that needs new technology,” she tells me. “It’s so expensive. Seriously. That expense is because of labour, and real-time rendering will cut your labour in half. Why should we pay so much for content any more? If Hollywood doesn’t wake up and get cheap, they’re going to get fossilised.”

Augmented reel-ality

Riva claims that facial mocap is only the start of how AR technology will affect the film industry. The traditional orders of post-production are going to be overhauled, she argues, with the onset of tools that let directors get polished, computer-generated visuals in real time. This ease of animation, she argues, means studios will have greater powers to adapt their performances on the fly; without the need to re-animate hours of footage if a tweak is needed along the way:

“You’re going to see a lot of film production lines; industrialised studio pipelines, that will be renovated by AR and real-time engines over the next two years. You’re going to go from factory-like production lines to more game-like, nonlinear productions.”

Last year, Unity’s CEO John Riccitiello touched on similar notes when he spoke about the scope for game worlds to be treated like reusable movie sets: “This is without doubt going to happen,” he told me at the time. “Someone will build a complete world. They’ll allow the world to age. They’ll shoot three movies in it and ten games in it. It’ll get reused, and edited, from Gotham City to something else.”

(Above: Some AR projects made in Unity. Credit: Unity)

“What is story in a persistent space that’s mixed with our everyday lives?”

Are games and films bleeding into each other? From speaking to the company’s representatives in Berlin, Unity seems adamant that AR will have a big role to play in how directors draw a connecting line between these virtual worlds and real-world actors. Not only will facial mapping mean quicker, cheaper animated performances, but directors will be able to use AR to overlay parts of a virtual filmset onto their real-world environment.

“It’ll be about informing a more traditional approach, like we do with virtual production now,” says Myhill. “It’s AR in a sense, but we’re still making a traditional film. That’s step one.

“In parallel, I think it’s going to raise the idea of ‘what is film?’ What is story in a persistent space that’s mixed with our everyday lives? We’re going to live in a data overlaid world, sharing our lives with persistent things in an AR space. We’re going to have friends and agents and these layers of story experiences… and they’re going to be integrated with our everyday lives.

“How amazing would it be to not watch, but be with characters?”

That, I imagine, would very much depend on the character.

Lead image credit: Dan Taylor