# Representation Learning 101 for Software Engineers

##### Figure 1: Overview of representation learning methods.
TLDR; Good representations of data (e.g., text, images) are critical for solving many tasks (e.g., search or recommendations). Deep representation learning yields state of the art results when used to create these representations. In this article, we review methods for representation learning and walk through an example using pretrained models.

## Introduction

Deep Neural Networks (DNNs) have become a particularly useful tool in building intelligent systems that simplify cognitive tasks for users. Examples include: neural search systems that identify the most relevant results given a natural language or image query, recommendation systems that provide personalized recommendations based on the user’s profile, and few-shot face verification systems.

The performance of many of these systems is gated on their ability to create representations (or features) of data with semantic meaning - numbers that truly encode the meaning of each data point with respect to the current task. For example, suppose a customer has a picture of an outfit they like. To enable the customer to find relevant clothes like it in our database, we need good measures of relevance that compare the user’s picture and all items in the database. Similarly, to recommend the right products to users based on their profiles, we also need high quality measures of relevance e.g., a measure of similarity between a user and all products.

DNN models are the tool of choice in realizing such systems, because they excel at learning semantically meaningful representations. However, across many real-world use cases, these models need to be carefully designed to fit both the task and data.

• What representation learning is, and why it’s valuable
• Methods for representation learning
• Learned Representations: how to evaluate good representations and when to use which methods

and share an example of implementation, using supervised representation learning for the task of semantic image search.

## Deep Representation Learning: What and Why!

Generally speaking, a good representation is one that makes a subsequent learning task easier. The choice of representation will usually depend on the choice of the subsequent learning task. [1](Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning, 2017).

Based on the above, we can define deep representation learning as training DNN models that yield numerical representations of data, suitable for solving a set of tasks.

To build intuition on why representation learning is valuable, we can review the question of how humans solve cognitive tasks. In many cases, we rely on heuristics - a set of fairly simple rules relevant to the task. For example, to identify if an image is a cat, we might perform the following checks: does it have four legs, two pointy ears, fur, whiskers, etc.? Based on the presence/absence of these salient features, we might determine with some level of confidence that it is, indeed, a cat.

Similarly, a neural network that succeeds at this same task should allocate its capacity (layers) such that it successfully translates (or disentangles) raw input data (e.g., image pixels) into a set of representations (e.g., eyes, ears, legs, whiskers) that are useful for the task.

##### Figure 3: Layers in a pretrained DenseNet121 model. a.) Layer 2 contains neurons that mostly respond to colors and simple textures. b.) Layer 424 contains neurons that respond to more complex, level concepts such as tree patterns and an eye. You can visually explore more representations learned by pretrained models here.

Keep in mind that layers within a DNN are stacked units of computation comprised of weights and bias terms, whose values are learned during training. Thus, if we formulate our training objective carefully, a DNN can yield representations that are then useful for a family of related tasks. Depending on the availability of labeled data, compute capacity, and distribution of data, there are several useful strategies for learning representations.

## Methods for Representation Learning

### Supervised Learning

When we train a DNN on a supervised learning task (e.g., classification), the training objective naturally yields representations that are useful for solving the task. In practice, this approach takes two main forms:

• Generic Classification: The most common examples of this are the use of models pretrained on the ImageNet dataset (1 million images, 100 classes) and, more recently, the Google BiT model trained on the JFT-300M dataset (300 million images, 18,291 classes)[13]. Extensive experiments have shown that the representations learned by these pretrained models (layers before the final softmax classifier layer) yield representations that are suitable for many other image processing tasks (transfer learning).
##### Figure 4: A 2D UMAP plot of features extracted using intermediate models constructed from a pretrained EfficientNetB0 model for a set of 200 natural images across 10 classes (arch, banana, Volkswagen beetle, Eiffel Tower, Empire State Building, Ferrari, pickup truck, sedan, Stonehenge, tractor). Given that the salient attributes for this specific set of natural images are high level features (e.g., wheels, doors), we see that the layers closest to the final classifier show the best performance, i.e., clean separation between classes.
• Metric Learning: Metric learning approaches aim to learn a good embedding space, such that the similarity between samples are preserved as distance between embedding vectors of the sample [9]. To this end, we can train DNN models with loss functions that yield this embedding space. Traditional metric loss functions include contrastive loss [8] and triplet loss [7], which are formulated to minimize intra-class distances and maximize inter-class distances.” One limitation here is that we frequently need to build training data pairs (consisting of positive and negative pairs), which can be challenging in practice. Furthermore, the performance of the model is sensitive to the strategies used for sampling these pairs [10]. To address this issue, variants of the softmax loss function (e.g., large margin softmax [11], angular softmax [12]), optimized for reducing the intra-class variation (i.e., making features of the same class compact) have been proposed, and yield state of the art results, with minimal complexity. Existing research also shows it is valuable to add L2 normalization prior to the softmax loss, as this more explicitly optimizes for cosine similarity [4][10]. Finally, researchers have also explored ensembling (training and combining results from multiple learners) [3] for even better performance, but with drawbacks in system complexity, and additional hyperparameters that need to be optimized.

Note: while much of the academic literature on metric learning focuses on applications in face recognition/verification, they can be adapted to other media modalities.

In general, a supervised approach assumes the availability of large labeled datasets - a requirement that is rarely achievable in practice. It also yields representations that may not generalize out-of-the-box, and may be susceptible to adversarial attacks.

Some applied examples include: Pinterest Pintext [14] - a multitask model trained using engagement data (clicks and repins) as labels (a measure of similarity between text queries); Visual metric learning at Pinterest - in which millions of Pinterest images with labels are used to learn a similarity metric for content-based image retrieval [9], [18]).

In this paradigm, the goal remains to train a supervised learning model, but find creative ways of automatically generating meaningful labels. Historically, this is done by designing pretext tasks - tasks that, if solved, yield semantically meaningful representations as a side effect. Some example pretext tasks used in the image domain include:

• Predicting the angle of rotation (geometric transformations)
• Predicting if two images with transformations (e.g., color transformations, or random crops) are the same (constrastive) [2]
• Predicting if two image patches come from the same image (image jigsaw puzzles) [16]
• Predicting the order of image frames from a video sequence (temporal)

For an extensive treatment on methods for self-supervised visual feature learning, see the survey paper [15].

Note that the self-supervised approach (commonly known as pretraining) is also well-known in the NLP field:

• Predicting the next word from a sequence of words
• Predicting the masked word in a sentence
• Given two sentences, predicting if one follows the other

A good pretext task is one which requires semantic understanding (or knowledge of important patterns) to solve (as seen in the examples above). The pretext task also takes advantange of known properties of the data in generating a training dataset; e.g., for the task of predicting the angle of rotation for an image, we can apply rotation transforms to images, and the labels are the rotation angles. If we construct pretext tasks that are inherently meaningful, then we can provide some signal (impose constraints) for the network to build notions of meaning. The harder the task, the more relevant the representations learned [16]. In general, a model is trained to solve the pretext task on a large dataset of unlabeled images, and the representation it learns is transferred to other downstream tasks.

Advances in self-supervised representation learning are particularly exciting. Recent research, which explores a contrastive formulation for pretext tasks, shows that self-supervised methods can achieve results at par with supervised learning! For example, SwAV[2] is only 1.2% below the performance of a fully supervised model.

### Unsupervised Learning

In some situations, it may be challenging to design good pretext tasks (e.g., data augmentation strategies or transforms are not meaningful, sampling negative or positive pairs is hard, etc.). For these situations, fully unsupervised methods can be explored. Many of these methods fall under the class of generative models, where the objective is to model a data distribution that can be subsequently sampled. Some examples of these models include an autoencoder, denoising autoencoder, VAE, and GANs (generation, inpainting, superresolution, colorization)[15]. While there is no explicit pretext task, the training objective still requires the model to disentangle input into salient features which may sometimes have semantic meaning. Strictly speaking, however, unsupervised learning is uncertain, and does not provide guarantees that the representations learned are particularly good.

## Learned Representations: Evaluation and Use

### Evaluation

What makes one representation better than another? One hypothesis is that an ideal representation is one in which the features within the representation correspond to the underlying causes of the observed data (with separate features or directions in featurespace corresponding to different causes), so that the representation disentangles the causes from one another. [1](Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning, 2017)

How do we know if we have good representations? How can we verify that a model has learned the underlying causal structures with respect to a task? This is hard to mathematically quantify, and historically has been evaluated based on downstream task performance (e.g., image classification, semantic segmentation, object detection, and action recognition) [15]. For example, representations learned using any of the methods listed above can be evaluated by using them as input for a linear logistic regression classifier on a downstream task. There have also been some interesting ideas put forth on ways to improve learned representations using clustering and pseudo labels; see [16], SwAV[2].

### When to Use What? Practical Considerations

Selecting an approach for representation learning is largely a tricky excercise in balancing effort (compute, time) and expected quality of representations. On the one hand, fully supervised methods yield the best performance for a specific task (e.g., see results from Pinterest [9]), but require the effortful curation of very large labeled datasets. On the other hand, fully unsupervised methods require no labeling effort, but do not provide strong guarantees for learning robust representations, resulting in varied performance.

The following high level notes are perhaps useful:

• Pretrained model baselines: Where possible, begin explorations using pretrained models. This may involve directly using features from a pretrained model (as seen in the next section) or finetuning the pretrained model, using a small amount of labeled data (transfer learning). Of course, this approach only works if the target task data distribution is similar to the data used to create the pretrained models.
• Self-supervised baselines: Where a large amount of unlabeled data exists, self-supervised methods are particularly useful. Explore pretext task strategies pertinent to the data in learning an initial set of representations, that can then be finetuned using pseudo labels [16] and/or a small amount of labeled data.

## Semantic Image Search with Pretrained Neural Network Models in Tensorflow

##### Figure 5: Screenshots from a web application we built which allows you to delve into semantic search query results, explore a visualization of embeddings, and perform live search. (Full source code here.)

To demonstrate how representations can be used for a concrete task, let us consider the task of semantic image search (also known as content-based retrieval, image retrieval, or reverse image search). We define semantic search as follows:

Given a dataset of existing images, and a new arbitrary image (query image), find a subset of images from the dataset that are most similar to the query image.

Here is a simplified implementation of semantic search outlined as a three-step process:

• Feature Extraction: First, a pretrained CNN model is used to extract features (vector representation) for each image in the dataset. Note that, depending on the layer within the model, we may have vectors of varied sizes. In the example snippet below, we load an EfficientNetB0 model from the keras model zoom, and use its last layer before (include_top=False) its linear classifier as a feature extractor. This yields a vector of size (1, 7, 7, 1280) or 62720 when flattened, per image.
 0 1 2 3 4 5 6 7  from tensorflow.keras.applications import EfficientNetB0 from tensorflow.keras.models import Model # ... model = EfficientNetB0(weights='imagenet', include_top=False) img_array = image_array_from_dir(dir_path) features = model.predict(img)
• Indexing: Once we have our feature vectors for each image, we need infrastructure ( an index) that enables us to efficiently store these representations, such that we can subsequently run search queries. As the size of the dataset becomes larger, two challenges emerge. First, a significant amount of space is needed to store vectors of size 62720 each, for millions of images. Second, it becomes computationally slow to exhaustively search over all images in our dataset (i.e., computing a similarity distance metric between the query and all dataset images and sorting results by distance). The first challenge can be addressed by applying transforms (e.g., dimensionality reduction, quantization) to our feature vectors. The second challenge can be addressed by using approximate nearest neighbour (ANN) methods [17]. Both of these solutions, while practical, result in some accuracy trade off. In the example below, we use the FAISS library which implements both exhaustive search and ANN search, as well as supports dimensionality reduction and quantization. (For a comparison of methods and libraries for ANN, see this blog post.)
 0 1 2 3 4 5 6  import faiss # ... feature_dim = features.shape[1] faiss_index = faiss.IndexFlatL2(feature_dim) faiss_index.add(features)
• Search: Finally, to return results for a search query, we retrieve the precomputed distance values between the searched image and all other images, sorted in the order of closest to farthest.
 0 1  k = 6 # number of similar items to return distances, indices = faiss_index.search(query, k) 

Full source code for the steps above is provided here.

## Conclusions

This article provides a high level overview of methods for representation learning and a simplified example of how good representations extracted using pretrained models can be used for a downstream task such as semantic image search. The reader is encouraged to explore the references provided for additional details on the theory/implementation for the methods discussed.

## References

[1] Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning. Vol. 1. Massachusetts, USA, MIT press, 2017. https://www.deeplearningbook.org/contents/representation.html

[2] Caron, Mathilde, et al. “Unsupervised Learning of Visual Features by Contrasting Cluster Assignments.” Advances in Neural Information Processing Systems 33 (2020).

[3] Opitz, Michael, et al. “Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly.” IEEE transactions on pattern analysis and machine intelligence (2018).

[4] Ranjan, Rajeev, Carlos D. Castillo, and Rama Chellappa. “L2-constrained Softmax Loss for Discriminative Face Verification.” arXiv preprint arXiv:1703.09507 (2017).

[5] Fefferman, Charles, Sanjoy Mitter, and Hariharan Narayanan. “Testing the Manifold Hypothesis.” Journal of the American Mathematical Society 29.4 (2016): 983-1049.

[6] Nguyen, Binh X., et al. “Deep Metric Learning Meets Deep Clustering: An Novel Unsupervised Approach for Feature Embedding.” arXiv preprint arXiv:2009.04091 (2020).

[7] Hoffer, Elad, and Nir Ailon. “Deep metric learning using Triplet network.” International Workshop on Similarity-Based Pattern Recognition. Springer, Cham, 2015.

[8] Chopra, Sumit, Raia Hadsell, and Yann LeCun. “Learning a Similarity Metric Discriminatively, with Application to Face Verification.” 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05). Vol. 1. IEEE, 2005.

[9] Zhai, Andrew, and Hao-Yu Wu. “Classification is a Strong Baseline for Deep Metric Learning.” arXiv preprint arXiv:1811.12649 (2018).

[10] Wang, Feng, et al. “Additive Margin Softmax for Face Verification.” IEEE Signal Processing Letters 25.7 (2018): 926-930.

[11] Liu, Weiyang, et al. “Large-Margin Softmax Loss for Convolutional Neural Networks.” ICML. Vol. 2. No. 3. 2016.

[12] Liu, Weiyang, et al. “Sphereface: Deep Hypersphere Embedding for Face Recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[13] Kolesnikov, Alexander, et al. “Big Transfer (BiT): General Visual Representation Learning.” arXiv preprint arXiv:1912.11370 6 (2019).

[14] Zhuang, Jinfeng, and Yu Liu. “PinText: A Multitask Text Embedding System in Pinterest.” Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.

[15] Jing, Longlong, and Yingli Tian. “Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey.” IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).

[16] Noroozi, Mehdi, et al. “Boosting Self-supervised Learning via Knowledge Transfer.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

[17] Johnson, Jeff, Matthijs Douze, and Hervé Jégou. “Billion-scale similarity search with GPUs.” IEEE Transactions on Big Data (2019).

[18] Zhai, A., Wu, H. Y., Tzeng, E., Park, D. H., & Rosenberg, C. (2019, July). Learning a Unified Embedding for Visual Search at Pinterest. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2412-2420).

Older

## Latest posts

##### Nov 15, 2022 · newsletter
November 2022 Perhaps November conjures thoughts of holiday feasts and festivities, but for us, it’s the perfect time to chew the fat about machine learning! Make room on your plate for a peek behind the scenes into our current research on harnessing synthetic image generation to improve classification tasks. And, as usual, we reflect on our favorite reads of the month. New Research! In the first half of this year, we focused on natural language processing with our Text Style Transfer blog series.
##### Nov 14, 2022 · post
by Introduction This post documents the first part of a research effort to quantify the impact of synthetic data augmentation in training a deep learning model for detecting manufacturing defects on steel surfaces. We chose to generate synthetic data using CycleGAN,1 an architecture involving several networks that jointly learn a mapping between two image domains from unpaired examples (I’ll elaborate below). Research from recent years has demonstrated improvement on tasks like defect detection2 and image segmentation3 by augmenting real image data sets with synthetic data, since deep learning algorithms require massive amounts of data, and data collection can easily become a bottleneck.
##### Oct 20, 2022 · newsletter
October 2022 We’ve got another action-packed newsletter for October! Highlights this month include the re-release of a classic CFFL research report, an example-heavy tutorial on Dask for distributed ML, and our picks for the best reads of the month. Open Data Science Conference Cloudera Fast Forward Labs will be at ODSC West near San Fransisco on November 1st-3rd, 2022! If you’ll be in the Bay Area, don’t miss Andrew and Melanie who will be presenting our recent research on Neutralizing Subjectivity Bias with HuggingFace Transformers.
##### Sep 21, 2022 · newsletter
September 2022 Welcome to the September edition of the Cloudera Fast Forward Labs newsletter. This month we’re talking about ethics and we have all kinds of goodies to share including the final installment of our Text Style Transfer series and a couple of offerings from our newest research engineer. Throw in some choice must-reads and an ASR demo, and you’ve got yourself an action-packed newsletter! New Research! Ethical Considerations When Designing an NLG System In the final post of our blog series on Text Style Transfer, we discuss some ethical considerations when working with natural language generation systems, and describe the design of our prototype application: Exploring Intelligent Writing Assistance.
##### Sep 8, 2022 · post
by This post has a companion piece: Ethics Sheet for AI-assisted Comic Book Art Generation I want to make a comic book. Actually, I want to make tools for making comic books. See, the problem is, I can’t draw too good. I mean, I’m working on it. Check out these self portraits drawn 6 months apart: Left: “Sad Face”. February 2022. Right: “Eyyyy”. August 2022. But I have a long way to go until my illustrations would be considered professional quality, notwithstanding the time it would take me to develop the many other skills needed for making comic books.
##### Aug 18, 2022 · newsletter
August 2022 Welcome to the August edition of the Cloudera Fast Forward Labs newsletter. This month we’re thrilled to introduce a new member of the FFL team, share TWO new applied machine learning prototypes we’ve built, and, as always, offer up some intriguing reads. New Research Engineer! If you’re a regular reader of our newsletter, you likely noticed that we’ve been searching for new research engineers to join the Cloudera Fast Forward Labs team.

### Popular posts

##### Oct 30, 2019 · newsletter
Exciting Applications of Graph Neural Networks
##### Nov 14, 2018 · post
Federated learning: distributed machine learning with data locality and privacy
##### Apr 10, 2018 · post
PyTorch for Recommenders 101
##### Oct 4, 2017 · post
First Look: Using Three.js for 2D Data Visualization
##### Aug 22, 2016 · whitepaper
Under the Hood of the Variational Autoencoder (in Prose and Code)
##### Feb 24, 2016 · post
"Hello world" in Keras (or, Scikit-learn versus Keras)

# Reports

In-depth guides to specific machine learning capabilities

# Prototypes

Machine learning prototypes and interactive notebooks

## ASR with Whisper

Explore the capabilities of OpenAI's Whisper for automatic speech recognition by creating your own voice recordings!

## NeuralQA

A usable library for question answering on large datasets.
https://neuralqa.fastforwardlabs.com

## Explain BERT for Question Answering Models

Tensorflow 2.0 notebook to explain and visualize a HuggingFace BERT for Question Answering model.