Machine learning continues to make its way into the arts, most recently in film and TV.
In a recent blog post, data scientists at 20th Century Fox and technical staff at Google Cloud described the approach they are using to predict audiences for their movies. (The tone of the post is fairly self-promoting, befitting the subject matter and industries involved.)
Their product, Merlin Video, is a deep learning tool that analyzes movie trailer videos. It was based on a pre-trained model - Google’s YouTube 8M video data set, which identifes objects in video, and tuned with movie trailers and marketing data from past movies. Fox released a paper describing the technology in detail. They have been using Merlin - with success - for a year and a half.
Merlin’s architecture (image source)
Fox’s software works to predict an audience, with clear implications for “testing” likely outcomes for certain types of films, which studios can use to market existing movies more effectively, and guide writers and directors toward making more commercially appealing stories. But can machines generate the stories themselves? Not yet - but this is not far off.
For a challenge, filmmaker Oscar Sharp and creative technologist Ross Goodwin used a neural network to create a screenplay. They created an LSTM recurrent neural network called Benjamin. They trained Benjamin with science fiction screenplays and prompted it with data from a science fiction filmmaking contest. Benjamin produced a screenplay, and Sharp and Goodwin, with a cast and crew, made a film, Sunspring from the screenplay with impressive and interesting results.
Clearly, Benjamin is not ready for commercial film, but the resulting film is surprisingly coherent (acknowledging that human minds will go a long way to find order in chaos).
Still, it’s clear there are useful applications of machine learning in entertainment, and we expect these products to improve, with interesting results.
More from the Blog
Dec 18 2018
We end 2018 with a round-up of some of the research, talks, sci-fi, visualizations/art, and a grab bag of other stuff we found particularly interesting, enjoyable, or influential this year (and we’re going to be a bit fuzzy about the definition of “this year”)! Research In addition to our own research, on recommendation engines, multi-task learning, and federated learning, we found three othe...
Dec 28 2018
by — 2018 was a fun and exciting year for natural language processing. A series of papers put forth powerful new ideas that improve the way machines understand and work with language. They challenge the standard way of using pretrained word embeddings like word2vec to initialize the first layer of a neural net, while the rest is trained on data of a particular task. Instead, these papers propose bet...
May 22 2019
by — Active learning allows us to be smart about picking the right set of datapoints for which to create labels. Done properly, this approach results in models that are trained on less data performing comparatively to models trained on much more data. In the world of meta-learning, we do not focus on label acquisition; rather, we attempt to build a machine that learns quickly from a small number of ...