Mar 29, 2019 · newsletter

Three exciting developments at TensorFlow Dev Summit 2019

The third annual TensorFlow Dev Summit was held. The entire first day of talks was live-streamed, and videos are available on the TensorFlow YouTube channel. I stayed up late (from the UK) watching a few of the talks, and here’s an entirely idiosyncratic description of three things I find exciting. A broader recap is provided by the TensorFlow team themselves.

TensorFlow Federated

TensorFlow Federated is a library for federated learning. Given the title of our last report, you already know we think this is an exciting area. It was a group of Google researchers who coined the term “federated learning” with the release of the Federated Averaging algorithm for learning from decentralized data, so it makes sense that Federated Learning is getting the TensorFlow treatment.

The library includes both the high-level “Federated Learning” API designed for wrapping existing TensorFlow models, and a low-level “Federated Core” API, which is intended to the development of new federated learning algorithms. The project is nascent, but particular attention has been paid to separating the various concerns of building a federated learning system. For instance, without some thought, it would easy to start entangling model operations and network communication. By providing high level abstractions, TensorFlow Federated allows experts in each area to contribute their respective knowledge, which could substantially lower the bar to entry for participating in federated learning research. The ability to wrap an existing Keras model is particularly appealing.

TensorFlow Federated currently includes only a simulated runtime (where all federated nodes actually live on the same, local, device), so it does not out-of-the-box “solve” federated learning. Eventually, other runtimes will be supported with the same API, meaning that the gap between researching novel federated approaches and deploying them to real federated networks is minimised.

TensorFlow Probability

Likewise to federated learning, probabilistic programming is also something we’ve written about at Fast Forward Labs. Since our probabilistic programming report was released, the ecosystem has evolved. PyMC and Stan both continue to develop, but perhaps the most notable change has been the emergence of so-called “deep probabilistic programming” libraries. These libraries combine the ability of deep neural networks to learn complex functions with the probabilistic paradigm of computing with distributions. runs atop PyTorch, and until recently, Edward was the go-to library for TensorFlow.

TF Probability takes a broader approach to probabilistic programming than Edward, introducing layers of abstraction into the “probprog” stack. At its base, it has complete interoperability with TensorFlow, which means all of TF’s linear algebra operations are available. On top of this is a set of “statistical building blocks” (their language): probability distributions and bijectors (composable transformations of random variables). There are also model building tools, including probabilistic programming language Edward2 - unsurprisingly, the successor to Edward - and a “probabilistic layers” component that can be used with high level APIs like Keras. Finally, there are a suite of probabilistic inference methods including Monte Carlo and Variational Inference.

The library has been evolving rapidly over the last year, but seems to be converging to a useful suite of tools. The TF Dev Summit demo of TensorFlow Probability included a simple and compelling walk through of building a regression model then including aleatoric (observation noise) and epistemic (model) uncertainty. It is encouraging to see probabilistic programming continue to mature and gain traction with existing software ecosystems.

Swift for TensorFlow

Anyone at CFFL will tell you this particular newsletter author has a bit of a soft spot for probabilistic programming. Despite developments in TF Probability, of all the developments announced at the TF Dev Summit, I find the progress in Swift for TensorFlow (which has moved into beta) the most promising.

Swift is an open source language from Apple, first designed to provide a better development experience for iOS and OSX (but now also supporting Linux, less a few niceties like automatic test discovery). Chris Lattner, who implemented much of the Swift language while at Apple, is now working on bringing TensorFlow to Swift. However, this is not a simple wrapper around the existing TensorFlow codebase, but rather a re-imagining of what tools for machine learning can be. Swift for TensorFlow integrates automatic differentiation into the Swift language itself, allowing easy definition of performant custom operations. Swift is fast, expressive and has an advanced type system. At CFFL, we love Python, and the Swift for TensorFlow team have also provided an almost native experience for interoperability with Python.

The watch words are “differentiable computing,” and we eagerly await more demo applications in this ecosystem. One advantage to Swift, being designed as a replacement for C family languages is that it is not necessary to wrap C libraries for performant numerical code. This means one can move up and down the numerical computing stack in the same language, which is also great for pedagogy. Speaking of pedagogy, Jeremy Howard of has noticed Swift too, and at the TF Dev Summit, it was announced that the next iteration of the course will include a Swift component from Chris Lattner himself.

Read more

Apr 2, 2019 · featured post
Mar 20, 2019 · featured post

Latest posts

Sep 22, 2021 · post

Automatic Summarization from TextRank to Transformers

by Melanie Beck · Automatic summarization is a task in which a machine distills a large amount of data into a subset (the summary) that retains the most relevant and important information from the whole. While traditionally applied to text, automatic summarization can include other formats such as images or audio. In this article we’ll cover the main approaches to automatic text summarization, talk about what makes for a good summary, and introduce Summarize. – a summarization prototype we built that showcases several automatic summarization techniques. more
Sep 21, 2021 · post

Extractive Summarization with Sentence-BERT

by Victor Dibia · In extractive summarization, the task is to identify a subset of text (e.g., sentences) from a document that can then be assembled into a summary. Overall, we can treat extractive summarization as a recommendation problem. That is, given a query, recommend a set of sentences that are relevant. The query here is the document, relevance is a measure of whether a given sentence belongs in the document summary. How we go about obtaining this measure of relevance varies (a common dilemma for any recommendation system). more
Sep 20, 2021 · post

How (and when) to enable early stopping for Gensim's Word2Vec

by Melanie Beck · The Gensim library is a staple of the NLP stack. While it primarily focuses on topic modeling and similarity for documents, it also supports several word embedding algorithms, including what is likely the best-known implementation of Word2Vec. Word embedding models like Word2Vec use unlabeled data to learn vector representations for each token in a corpus. These embeddings can then be used as features in myriad downstream tasks such as classification, clustering, or recommendation systems. more
Jul 7, 2021 · post

Exploring Multi-Objective Hyperparameter Optimization

By Chris and Melanie. The machine learning life cycle is more than data + model = API. We know there is a wealth of subtlety and finesse involved in data cleaning and feature engineering. In the same vein, there is more to model-building than feeding data in and reading off a prediction. ML model building requires thoughtfulness both in terms of which metric to optimize for a given problem, and how best to optimize your model for that metric! more
Jun 9, 2021 ·

Deep Metric Learning for Signature Verification

By Victor and Andrew. TLDR; This post provides an overview of metric learning loss functions (constrastive, triplet, quadruplet, and group loss), and results from applying contrastive and triplet loss to the task of signature verification. A complete list of the posts in this series is outlined below: Pretrained Models as Baselines for Signature Verification -- Part 1: Deep Learning for Automatic Offline Signature Verification: An Introduction Part 2: Pretrained Models as Baselines for Signature Verification Part 3: Deep Metric Learning for Signature Verification In our previous blog post, we discussed how pretrained models can serve as strong baselines for the task of signature verification. more
May 27, 2021 · post

Pretrained Models as a Strong Baseline for Automatic Signature Verification

By Victor and Andrew. Figure 1. Baseline approach for automatic signature verification using pretrained models TLDR; This post describes how pretrained image classification models can be used as strong baselines for the task of signature verification. The full list of posts in the series is outlined below: Pretrained Models as Baselines for Signature Verification -- Part 1: Deep Learning for Automatic Offline Signature Verification: An Introduction Part 2: Pretrained Models as Baselines for Signature Verification Part 3: Deep Metric Learning for Signature Verification As discussed in our introductory blog post, offline signature verification is a biometric verification task that aims to discriminate between genuine and forged samples of handwritten signatures. more

Popular posts

Oct 30, 2019 · newsletter
Exciting Applications of Graph Neural Networks
Nov 14, 2018 · post
Federated learning: distributed machine learning with data locality and privacy
Apr 10, 2018 · post
PyTorch for Recommenders 101
Oct 4, 2017 · post
First Look: Using Three.js for 2D Data Visualization
Aug 22, 2016 · whitepaper
Under the Hood of the Variational Autoencoder (in Prose and Code)
Feb 24, 2016 · post
"Hello world" in Keras (or, Scikit-learn versus Keras)


In-depth guides to specific machine learning capabilities


Machine learning prototypes and interactive notebooks


A usable library for question answering on large datasets.

Explain BERT for Question Answering Models

Tensorflow 2.0 notebook to explain and visualize a HuggingFace BERT for Question Answering model.

NLP for Question Answering

Ongoing posts and code documenting the process of building a question answering model.

Interpretability Revisited: SHAP and LIME

Explore how to use LIME and SHAP for interpretability.


Cloudera Fast Forward is an applied machine learning reseach group.
Cloudera   Blog   Twitter