This facial recognition system tracks how you’re enjoying a movie

@techcrunch / 6:19 pm PDT • July 25, 2017

Image Credits:

As moviemaking becomes as much a science as an art, the moviemakers need ever-better ways to gauge audience reactions. Did they enjoy it? How much… exactly? At minute 42? A system from Caltech and Disney Research uses a facial expression tracking neural network to learn and predict how members of the audience react, perhaps setting the stage for a new generation of Nielsen ratings.

The research project, just presented at IEEE’s Computer Vision and Pattern Recognition conference in Hawaii, demonstrates a new method by which facial expressions in a theater can be reliably and relatively simply tracked in real time.

It uses what’s called a factorized variational autoencoder — the math of it I am not even going to try to explain, but it’s better than existing methods at capturing the essence of complex things like faces in motion.

The team collected a large set of face data by recording audiences of hundreds watching movies (Disney ones, naturally). An infrared hi-def camera captures everyone’s motions and faces, and the resulting data — 16 million or so data points — was fed to the neural network.

Once it had finished training, the team set the system on watching audience footage in real time and attempting to predict the expression a given face would make at various points. They found that it took about 10 minutes to warm up to the audience, so to speak, and after that reliably predicted laughs and smiles (no crying or jump scares tracked yet, it seems).

Of course, this is just one application of a technology like this — it could be applied in other situations like monitoring crowds, or elsewhere interpreting complex visual data in real time.

“Understanding human behavior is fundamental to developing AI systems that exhibit greater behavioral and social intelligence,” said Caltech’s Yisong Yue in a news release. “For example, developing AI systems to assist in monitoring and caring for the elderly relies on being able to pick up cues from their body language. After all, people don’t always explicitly say that they are unhappy or have some problem.”

This facial recognition system tracks how you’re enjoying a movie

Latest Stories

Hugging Face releases a benchmark for testing generative AI on health tasks

Tesla still plans to build 1,800-mile charging corridor for semi trucks despite Biden funding snub

Robots can make jobs less meaningful for human colleagues

Ibotta’s IPO opens sharply higher, hinting at warming public-market interest in tech shares

Lacework, last valued at $8.3B, is in talks to sell for just $150M to $300M, say sources

Hackers are threatening to publish a huge stolen sanctions and financial crimes watchlist