Sep 8, 2022 · post

Thought experiment: Human-centric machine learning for comic book creation

I want to make a comic book. Actually, I want to make tools for making comic books. See, the problem is, I can’t draw too good. I mean, I’m working on it. Check out these self portraits drawn 6 months apart:

Left: “Sad Face”. February 2022. Right: “Eyyyy”. August 2022.

But I have a long way to go until my illustrations would be considered professional quality, notwithstanding the time it would take me to develop the many other skills needed for making comic books.

So that’s why I want to make tools for making comic books. What kind of tools, you ask? Well, I have an admittedly naive understanding of how to make comic books, but here are some ideas. They aren’t perfect, but I believe something contemplated is closer to reality than the uncontemplated.

Concept artificer

What if you could do something like this?

Or, more generally:

It’s not far-fetched. Here’s a picture from a 2022 paper called “CLIPstyler: Image Style Transfer With a Single Text Condition,”1:

This immediately suggests the following method for making my comic book:

I started this drawing at 4:12 pm and finished it at 4:46 pm, taking 34 minutes.

The framework is simple: I draw my entire comic book with whatever art I can muster, then I can use a style transfer model to make it look “good”. If I can crank out 2 pages per hour, then a 30-page comic book would take a mere 15 hours of my labor.2

But it’s easy to pick this simplistic framework apart. For example, one would like stylistic consistency of the same subjects from page to page, a desirable property that is analogous to temporal consistency in video processing. A style transfer model is unlikely to deliver such consistency for a comic book out of the box.

What exactly is temporal consistency?

Consider if you wanted to colorize a black-and-white video. Videos can be thought of as an ordered sequence of images or frames, so one way to do it would be to colorize each frame independently using your favorite single-image technique. The problem with that approach is that a single-image technique might assign different colors to the same subject in two successive frames, resulting in flickering as the colors quickly change.

Temporal consistency refers to the absence of such artifacts. Now take a moment to watch the cool video at “Learning Blind Video Temporal Consistency”3, come back, squint your eyes, and imagine how you could apply this to comic book page consistency. Comic books, like videos, are also an ordered sequence of images - although of course there are key differences, primarily that successive video frames typically depict the same scene fractions of a second apart, whereas the scenes depicted in successive comic book panels are likely more sparse in time.

Exploiting prior beliefs to promote consistency

Comic books are information rich, and the prior beliefs we have about them can be exploited in the context of optimization. Let’s frame the problem using an example 3-panel comic:

What are the repeated elements where stylistic consistency would be desired? Let’s list some out:

Elements where stylistic consistency is desired
Characters Settings
The Hero The Hero's home
The Princess The Princess' castle
The Dragon

The character of the Hero appears in every panel, and if we want to visually communicate that it is the same character in each panel, we should ensure they look the same. Similarly the setting of the Hero’s home appears both in the first and last panel. The other listed elements appear only once, but it’s easy to imagine a more detailed story where these elements appear more than once, and then stylistic consistency becomes a concern for them as well.

There is not one single or obviously most effective way to encode prior belief, but one very basic way is to ask whether a desired element simply appears in a panel. We can express our prior belief in terms of conditional probabilities about images:

$$ \hat{p}(E_i|X_j) := \hat{p}_{ij} := \begin{cases} 1 \text{ if } E_i \text{ appears in } X_j, \
0 \text{ otherwise } \end{cases} $$

Where Ei represents an element such as the Hero or the Hero’s home, and Xj the jth panel.

Detecting the appearance of an element in a panel is not a trivial computer vision problem - for example a character’s location in the panel, pose, and all manner of details are not fixed. But image classification is a well-studied problem to which deep learning has been applied with great success. If we presuppose a good style transfer network and a family of binary classifiers that can estimate the conditional probability that an element Ei appears in an image Xj (pij for brevity) then we immediately have a comic book making framework that incorporates this notion of stylistic consistency:

And we can make our comic book by optimizing the Comic Book Loss over all panels with respect to the style transfer model parameters using the usual methods:

$$ \begin{array} \
C_{ij} := H(\hat{p_{ij}}, p_{ij}) = -\hat{p_{ij}}\log{p_{ij}} - (1 - \hat{p_{ij}})\log{(1 - p_{ij})} \
\text{Comic Book Loss} := \sum\limits_{j} S_j + \sum\limits_{i,j} C_{ij} \end{array} $$

Where the terms Cij are defined as the binary cross-entropy between the distribution encoding our prior belief about the panels, and the estimated distribution from our classifiers. These terms are minimized when our stylistic classifiers are very confident that their respective subject appears or doesn’t appear in the desired panels, according to our prior beliefs. The terms Sj are simply whatever losses our style transfer network typically uses.

In summary, here’s how I’m going to make my comic book:

  1. Draw concept art for all my pages.
  2. Identify elements where I want stylistic consistency.
  3. Train binary classifiers for each such element.
  4. Fine-tune a good style transfer model to minimize the Comic Book Loss over my concept art.
  5. Voila!

Ok, but why?

Honestly, I really do want to make comic books!

Alright, I have some additional motivations.

For one, I’m obsessed with fiction. Fiction is the funhouse mirror reflection of real life, and it has often been my welcome guide through a confusing world by helping me more clearly imagine what is possible. I know I’m not alone in feeling that way. My hope is that developing systems like the one outlined above will make a storyteller out of someone who wouldn’t have otherwise been one, for lack of artistic skill, or time, or whatever. For me personally, making a high quality comic book I could be proud of could take years if I do it the traditional way4, but perhaps I can leverage my already-accumulated years of experience with machine learning to do it more quickly.

For another, as a machine learning practitioner, I think a lot about the implications of the technology I develop on society, because I want to create systems that I can be proud of. It’s easy to find many credible opinions that technological development is not an unqualified good thing. One criticism levied at AI algorithms in particular is that the automation they’ll enable will destroy jobs for humans. I want to put forth my idea above as a partial answer to that - here’s a new application5, with the potential to become a new kind of job. Moreover it’s one with human involvement baked into the very core of the process - superficially in order to produce concept art, but more fundamentally because AI and machine learning can’t tell us what stories we want to make!


There’s still a lot we could improve about the above framework, notwithstanding the challenge of training image classifiers for important story elements. For example, we could make our notion of consistency more precise by asking not only if an element simply appears in a panel at all, but asking how closely it matches the location and content of the corresponding element in the concept art. And if we have some thoughts or directions about a panel expressed as text, it would be great if the final output could be conditioned on that as well. I’d love to hear your thoughts, so drop me a line!

  1. Kwon, Gihyun, and Jong Chul Ye. “CLIPstyler: Image Style Transfer With a Single Text Condition,” 18062–71, 2022. ↩︎

  2. Equivalent to 2 uncomfortable working days, or 5 great working days. And please take this estimate with a grain of salt. ↩︎

  3. Lai, Wei-Sheng, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. “Learning Blind Video Temporal Consistency,” 170–85, 2018. ↩︎

  4. Not least because I’d have to learn a hell of a lot of new skills. ↩︎

  5. I am not the first to think of using AI for comic books, although I believe my formulation of the task is novel. See Yang, Xin, Zongliang Ma, Letian Yu, Ying Cao, Baocai Yin, Xiaopeng Wei, Qiang Zhang, and Rynson W. H. Lau. “Automatic Comic Generation with Stylistic Multi-Page Layouts and Emotion-Driven Text Balloon Generation.” arXiv, January 26, 2021.
    See also ↩︎

Read more

Nov 14, 2022 · post
Jul 29, 2022 · post

Latest posts

Nov 15, 2022 · newsletter

CFFL November Newsletter

November 2022 Perhaps November conjures thoughts of holiday feasts and festivities, but for us, it’s the perfect time to chew the fat about machine learning! Make room on your plate for a peek behind the scenes into our current research on harnessing synthetic image generation to improve classification tasks. And, as usual, we reflect on our favorite reads of the month. New Research! In the first half of this year, we focused on natural language processing with our Text Style Transfer blog series. more
Nov 14, 2022 · post

Implementing CycleGAN

by Michael Gallaspy · Introduction This post documents the first part of a research effort to quantify the impact of synthetic data augmentation in training a deep learning model for detecting manufacturing defects on steel surfaces. We chose to generate synthetic data using CycleGAN,1 an architecture involving several networks that jointly learn a mapping between two image domains from unpaired examples (I’ll elaborate below). Research from recent years has demonstrated improvement on tasks like defect detection2 and image segmentation3 by augmenting real image data sets with synthetic data, since deep learning algorithms require massive amounts of data, and data collection can easily become a bottleneck. more
Oct 20, 2022 · newsletter

CFFL October Newsletter

October 2022 We’ve got another action-packed newsletter for October! Highlights this month include the re-release of a classic CFFL research report, an example-heavy tutorial on Dask for distributed ML, and our picks for the best reads of the month. Open Data Science Conference Cloudera Fast Forward Labs will be at ODSC West near San Fransisco on November 1st-3rd, 2022! If you’ll be in the Bay Area, don’t miss Andrew and Melanie who will be presenting our recent research on Neutralizing Subjectivity Bias with HuggingFace Transformers. more
Sep 21, 2022 · newsletter

CFFL September Newsletter

September 2022 Welcome to the September edition of the Cloudera Fast Forward Labs newsletter. This month we’re talking about ethics and we have all kinds of goodies to share including the final installment of our Text Style Transfer series and a couple of offerings from our newest research engineer. Throw in some choice must-reads and an ASR demo, and you’ve got yourself an action-packed newsletter! New Research! Ethical Considerations When Designing an NLG System In the final post of our blog series on Text Style Transfer, we discuss some ethical considerations when working with natural language generation systems, and describe the design of our prototype application: Exploring Intelligent Writing Assistance. more
Sep 8, 2022 · post

Thought experiment: Human-centric machine learning for comic book creation

by Michael Gallaspy · This post has a companion piece: Ethics Sheet for AI-assisted Comic Book Art Generation I want to make a comic book. Actually, I want to make tools for making comic books. See, the problem is, I can’t draw too good. I mean, I’m working on it. Check out these self portraits drawn 6 months apart: Left: “Sad Face”. February 2022. Right: “Eyyyy”. August 2022. But I have a long way to go until my illustrations would be considered professional quality, notwithstanding the time it would take me to develop the many other skills needed for making comic books. more
Aug 18, 2022 · newsletter

CFFL August Newsletter

August 2022 Welcome to the August edition of the Cloudera Fast Forward Labs newsletter. This month we’re thrilled to introduce a new member of the FFL team, share TWO new applied machine learning prototypes we’ve built, and, as always, offer up some intriguing reads. New Research Engineer! If you’re a regular reader of our newsletter, you likely noticed that we’ve been searching for new research engineers to join the Cloudera Fast Forward Labs team. more

Popular posts

Oct 30, 2019 · newsletter
Exciting Applications of Graph Neural Networks
Nov 14, 2018 · post
Federated learning: distributed machine learning with data locality and privacy
Apr 10, 2018 · post
PyTorch for Recommenders 101
Oct 4, 2017 · post
First Look: Using Three.js for 2D Data Visualization
Aug 22, 2016 · whitepaper
Under the Hood of the Variational Autoencoder (in Prose and Code)
Feb 24, 2016 · post
"Hello world" in Keras (or, Scikit-learn versus Keras)


In-depth guides to specific machine learning capabilities


Machine learning prototypes and interactive notebooks

ASR with Whisper

Explore the capabilities of OpenAI's Whisper for automatic speech recognition by creating your own voice recordings!


A usable library for question answering on large datasets.

Explain BERT for Question Answering Models

Tensorflow 2.0 notebook to explain and visualize a HuggingFace BERT for Question Answering model.

NLP for Question Answering

Ongoing posts and code documenting the process of building a question answering model.

Cloudera Fast Forward Labs

Making the recently possible useful.

Cloudera Fast Forward Labs is an applied machine learning research group. Our mission is to empower enterprise data science practitioners to apply emergent academic research to production machine learning use cases in practical and socially responsible ways, while also driving innovation through the Cloudera ecosystem. Our team brings thoughtful, creative, and diverse perspectives to deeply researched work. In this way, we strive to help organizations make the most of their ML investment as well as educate and inspire the broader machine learning and data science community.

Cloudera   Blog   Twitter

©2022 Cloudera, Inc. All rights reserved.