Hollywood and the academic artificial intelligence community have an interesting relationship, to say the least. Many people interested in the field, myself included, love science fiction, but regularly end up being the obnoxious commentator on family movie night pointing out that Johnny Depp’s Transcendence is more anti-intelligence than artificial intelligence. So it wasn’t all that surprising that researchers were a bit perplexed when actress and director Kristen Stewart appeared as an author on a machine learning paper.
Stewart directed Come Swim, a short film about a man’s day told through both impressionism and realism. The film features the use of the so-called “style transfer” technique to build its story.
Style transfer is the fun technique that involves using convolutional neural networks to artistically alter video in real time. To do this, you basically show your algorithm an image (say a painting by van Gogh) and then teach it to apply the same painting technique to any regular photo.
The short, three-page paper, entitled Bringing Impressionism to Life with Neural Style Transfer in Come Swim, is really just a high-level case study about what it’s like to apply style-transfer techniques to cinema.
In addition to Stewart, Bhautik Joshi and David Shapiro of Adobe and Starlight Studios, respectively, speak to the challenges of trying to reign in style transfer for controlled use. The team’s approach comes off a bit like blindfolded dart-throwing, but, to be fair, the trial and error process and dealing with trade-offs is pretty much what machine learning research is like at all levels.
One would almost expect that artists and movie-makers would be more comfortable with improvisation than researchers, but this doesn’t seem to be the case. “In a production setting, however, a great deal of creative control is needed to tune the result, and a rigid set of algorithmic constraints run counter to the need for this creative exploration,” note the authors of the paper.
This isn’t the first time that style transfer has been used for artistically expressive purposes. Some in the research community were quick to point out that a paper was written recently about the movie Interstellar. And on the other side of the table, style-transfer techniques have been used for lower-profile artistic videos in the past.
The prospect is none-the-less interesting and it’s likely we will see more of style transfer and machine learning in general in film making. Machines have already produced music, movie trailers and scripts. So while it seems like humans still remain on top, just ask yourself, how long is it until machines can create something better than [insert your favorite bad movie here].