The promise of Machine Learning (ML) on edge devices holds potential to enable new capabilities while reaping the benefits associated with on-device computation. As an environment that is frequently the source of data (user interactions, sensors such as cameras, acelerometers, etc.), the browser within PCs, mobile and IoT devices represents an important edge “platform.” Deploying models in such environments can improve latency for interactive applications, reduce model distribution costs, and enable privacy as data is no longer sent to remote servers for analysis.
Why Machine Learning in the Browser?
By bringing computation directly to the user, there is opportunity to improve application latency. This is especially true for tasks that may or may not be computationally intensive but have been traditionally performed on remote servers. For example, recommendation tasks such as predicting a user’s next action (that previously involved roundtrip calls to a remote server (upload user data, process, download result)) can be performed in the browser with faster response times.
Perhaps, one of the most compelling strategies for building user trust in a product is to offer objective notions of privacy. In many cases, systems designed to be provably private (“we do not store your data”) can be more convincing than systems which offer privacy assurances (“we store your data, but only use it for specific known purposes”). With ML in the browser, it is possible to perform analysis on a client’s data on their device via the browser and provide value to them without transmitting their data to any servers for processing. An example is the ability to offer language translation or image captioning or document classification services for sensitive content in the browser. Within the same framework, models can be trained on a user’s local data and used to provide improved, personalized services.
Setting up a traditional environment for running ML models can be tedious and challenging to deploy. While these issues can now be addressed with the use of docker containers as well as dependency management packages, it is still challenging to package and distribute applications with ML models on end user devices. The process of bundling a client application that first installs frameworks such as Tensorflow or Pytorch, simply for the purpose of enabling ML capabilities, can be error prone, and varies with the distribution platform.
On the other hand, browser based deployments can leverage the standardized web environment with zero installation costs for the user and distribution costs for the developer. Costs associated with building for multiple platforms (e.g., CPU architectures, Operating Systems) are also avoided as mature web standards allow web applications to remain stable across these environments.
Libraries for Machine Learning in the Browser: Tensorflow.js
As of right now, Tensorflow.js appears to be the most mature API in terms of maintenance, integration with the broader ML ecosystem, and community adoption. TensorFlow.js consists of two sets of APIs: the Ops API which provides lower-level linear algebra operations (e.g., matrix multiplication, tensor addition, etc.), and the Layers API, which provides higher-level model building blocks and best practices with emphasis on neural networks. The Layers API is modeled after the tf.keras namespace in TensorFlow Python, which is based on the widely adopted Keras API and supports rapid prototyping of complex ML models.
Figure shows the architecture of the Tensorflow.js library. source
TensorFlow.js is designed to run in-browser and server-side, as shown in the figure above. When running inside the browser, it utilizes the GPU of the device via WebGL to enable fast parallelized floating point computation. In Node.js, TensorFlow.js binds to the TensorFlow C library, enabling full access to TensorFlow. TensorFlow.js also provides a slower CPU implementation as a fallback (omitted in the figure for simplicity), implemented in plain JS. This fallback can run in any execution environment and is automatically used when the environment has no access to WebGL or the TensorFlow binary. (See the Tensorflow.js paper here.)
Tensorflow.js integrates well with the broader Tensorflow ecosystem by providing a converter which also allows users to convert, load, and run existing Tensorflow models. Developers can build, train, optimize and test their models in Tensorflow (Python) and then export the resulting model for use in the browser.
Current Applications and Outlook
With Tensorflow.js, users can prototype and run models of varying complexity from simple linear/polynomial regression to complex neural networks for image recognition, object detection, pose detection, and Natural Language Processing.
There have also been interesting applications ranging from prototyping handsfree interactions for increased accessiblity, hand gesture interaction and numerous examples for teaching ML concepts. Furthermore, and as we mention in an earlier report, Tensorflow.js can enable Federated Learning where locally trained models (not data) from participating devices are shared with a central server. The server combines these into an updated federated model which is then shared with each device.
As edge devices become more compute capable, it is likely that more processing, including model training and inference tasks, will be shifted to the browser. It is also likely that more third party libraries and design patterns will begin to integrate the use of ML models in the browser.
More from the Blog
Jan 29 2019
by — UMAP explorer: an interactive visualization of the MNIST data set We’re in the middle of work on our next report, Learning with Limited Labeled Data, and the accompanying prototype. For the prototype’s front-end we wanted to be able visualize and explore the embedding of a large image data set. Once you get into the tens of thousands of points, this can be a challenge to do in the browser. T...
Feb 28 2019
by — Judea Pearl, the inventor of Bayesian networks, recently published a book called The Book of Why: The New Science of Cause and Effect. The book covers a great many things, including a detailed history of how the fields of causality and statistics have long been at odds, Pearl’s own do-calculus framework for teasing causal inferences from observational data, and why (in Pearl’s view) the future ...
Jul 22 2019
by — We discussed this research as part of our virtual event on Wednesday, July 24th; you can watch the replay here! Convolutional Neural Networks (CNNs or ConvNets) excel at learning meaningful representations of features and concepts within images. These capabilities make CNNs extremely valuable for solving problems in the image analysis domain. We can automatically identify defects in manufactur...