Generative Adversarial Networks (GANs) wowed the world in 2014 with their ability to generate what we considered to be realistic images. While these images were quite low resolution, researchers kept working on how to perfect these methods in order to increase the quality of the images and even to apply the algorithm on other types of data like text and sound.
However, until recently there has been little success in making realistic videos. The main problem with making videos is temporal consistency: while people can be forgiving in one frame and find some interpretation for unrealistic regions, we are adept at seeing inconsistencies with how videos progress.
For example, we can accept some strange looking texture in the background of an image as simply some strange looking background. However, if that background is randomly changing from frame to frame in a video, we immediately discount the video. It is exactly this temporal consistency which has plagued researchers trying to apply GANs to videos – while each frame seemed realistic taken on its own, when assembled into a video, there were considerable inconsistencies which ruined any illusion of realism. This restricted the ability to reuse models that showed success at generating individual images, and forced researchers to come up with new methods to deal with the temporal nature of videos.
Recently, researchers at NVIDIA and MIT have come up with a new type of GAN, vid2vid, which primarily addresses this problem by explicitly incorporating how things seem to be moving within the video, in order to continue this motion in future frames. (In addition, they follow previous work, which uses a multi-resolution approach for generating high resolution images). This is done by calculating the optical flow of the image, which is a classic computer vision method that simply has not been incorporated into such a model until now.
The results are quite staggering (we highly recommend watching their release video). With the model you can create dashboard camera footage from the initial segmentation frame (allowing you to change the type and shape of objects in the frame by simply drawing in the corresponding color); it’s even possible to create realistic looking dance videos from pose information. It’s interesting to see this new method as compared with previous methods, to really get a sense of how important this additional temporal information is for making realistic results.
These high quality results are quite exciting and are groundbreaking work in the field of video generation. From applications in generating synthetic training data to use in creative projects, the vid2vid model itself is instantly applicable.
Even more interesting is how the field as a whole will learn from this research and start finding ways to incorporate other classic algorithms into neural networks. Just as conv-nets explicitly encoded the two dimensional understanding we have for images into models so that they can more quickly and accurately learn how to work with that data, this method explicitly encodes our understanding of how frames of a video flow from one to another (albeit this was much trickier to do than the conv-net example!). We’re interested in seeing what other algorithms will be incorporated into neural networks like this and what capabilities these models will have.
More from the Blog
Sep 28 2018
by — Machine learning on Spark: an abridged history Apache Spark - the cluster computing framework that’s been throwing shade at MapReduce since 2011 - has always been a bit remarkable, because it bridged the divide between data engineering and data science. One of the great promises of Spark was that it would be easy, trivial almost, to scale machine learning and data science to arbitrarily l...
Sep 28 2018
by — Machine learning (and more specifically, deep learning) techniques are recognized for their ability to obtain high accuracy for a variety of classification problems. Deep learning models are more complex than most traditional models and often have thousands of parameters and commensurately require more labeled training data. Obtaining this data can be an expensive and time-consuming process, ...
Dec 6 2018
by — Our prototypes are designed to demonstrate the value of the technologies we research. For our most recent prototype, Turbofan Tycoon, we decided that the best way to demonstrate the value of federated learning was to place you in an interactive simulation where you’re in charge of maintaining four turbofan engines. In this post, I’m going to try and explain a bit about why we decided that, an...