Resources
Feb 6, 2018 |
59 min
|
99.7k views
|

Machine Learning with R and TensorFlow

J.J. Allaire's keynote at rstudio::conf 2018 on the R interface to TensorFlow (https://tensorflow.rstudio.com), a suite of packages that provide high-level interfaces to deep learning models (Keras) and standard regression and classification models (Estimators), as well as tools for cloud training, experiment management, and production deployment. The talk also discusses deep learning more broadly (what it is, how it works, and where it might be relevant to users of R in the years ahead).

Slides: https://beta.rstudioconnect.com/ml-with-tensorflow-and-r/
JJ Allaire: - https://github.com/jjallaire
Twitter: @fly_upside_down https://twitter.com/fly_upside_down
Related blog post: https://blog.rstudio.com/2018/02/06/tensorflow-for-r/

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Thank you very much, Hadley, and thank all of you for coming to the conference. I hope you had a great first few days and great night last night. It's a great privilege for us at RStudio to be here with all of you this week.

Today I'm going to talk about machine learning with TensorFlow and R. For those of you who don't know, TensorFlow is an open source project from Google. It actually came out just a little bit over two years ago. And from the day it came out, I have been excited about what we could do with TensorFlow from R. And for the last about 18 months, me and some others at RStudio have been working on R interfaces to TensorFlow, and I'm really excited today to share a lot of that work with all of you.

So I'm going to start by giving an introduction to kind of what is TensorFlow, what are the core constituent parts of TensorFlow, and how does it work at kind of a low level. The principal application, but by no means the exclusive application of TensorFlow is for deep learning. So I want to talk a little bit conceptually what is deep learning, how does it work, what is it useful for, not useful for. And then I want to get into the work we've done in R to provide interfaces to TensorFlow for doing both deep learning and for other types of machine learning.

What is TensorFlow?

So let's start with what is TensorFlow. And you might offhand just say, oh, TensorFlow, that's a deep learning library. It is that, but it's actually considerably more than that. TensorFlow is actually a very general purpose numerical computing library. And in R, we have a long history of providing interfaces to numerical computing libraries. The original S language was the motivator for it was to provide interfaces to Fortran numerical computing libraries. And today in R, we wrap that code, we wrap code from Blasses, we wrap code from the Eigen C++ library, the Armadillo C++ library. In R, we love numerical computing libraries, and we love creating lovely interfaces to them from R.

So this is a new numerical computing library, and it has some interesting attributes. First of all, it's open source, as I said before. Second of all, it's actually hardware independent. So a TensorFlow program can run equally well on a CPU and take advantage of all the cores on a CPU, and that's actually using the Eigen and Blass libraries that we already use from R. But it can also run on a GPU or multiple GPUs. And there's even a thing that Google has created called a TPU, a tensor processing unit, which is hardware that is designed to run TensorFlow programs. And there will actually be other instances of this in the future. So hardware independence is a really cool attribute of TensorFlow.

Another really cool attribute of it for numerical computing is that it supports automatic differentiation. And that's used extensively in the deep learning parts of TensorFlow. The other thing is it was built from the ground up for deployment, and scalable deployment, so it supports distributed execution, it supports very, very large datasets. So it's a really cool numerical computing library that we can do a lot of things with in R.

And that's kind of the first bullet point of why our users should care. New numerical computing library, what can we do with it? One of the other cool things about it is that there are many built-in optimization algorithms that don't require that all the data is in RAM. So typically, the optimization algorithms we use require that we manage to get the whole dataset in RAM. But now we can have a 10 gigabyte dataset, and we actually feed data to models just a small batch at a time, and the optimizer will work correctly. So it lets us do modeling and machine learning on very large datasets without having to have a huge amount of RAM.

Another piece of TensorFlow that's interesting is that typically when we think about building a model in R and then we want to deploy it, we actually need to bring R code along with it in the deployment. But the whole design of TensorFlow is that when you build a TensorFlow model, you can deploy it entirely separate. You don't need R or Python or any other code. It's just a C++ runtime. So this is actually really cool, too. So the last piece, I think we're really good in R at writing interfaces to all kinds of things in computing, and I think we can do a lot of wonderful things with R interfaces to TensorFlow.

Tensors and the TensorFlow graph

So some of the basics of TensorFlow, where does that name TensorFlow, what are tensors, what's flowing, just to give you a baseline of what we're talking about here before we get into talking about deep learning. So tensors, actually everyone here already works with tensors pretty much all the time. Tensors are just multidimensional arrays. So in R, the core data type vector and matrices, those are our tensors. So a one-dimensional tensor is just an R vector, a two-dimensional tensor is an R matrix, and then you get into 3D and 4D arrays, which R also supports. A 0D tensor is a scalar, which R does not have a scalar data type, but you can think of it conceptually as a vector that's always of length one. So you're going to be dealing with tensors, and the good news is we already deal with tensors all the time. So there's really nothing new to learn there.

I want to give some examples of tensor data, kind of how it plays out in TensorFlow. And one of the things to remember, if you think about a data frame, each row is an observation. So there's always a dimension that's dedicated to observations or samples. So in the case of a 2D tensor, I can just take a data frame, turn it into a matrix, and now I've got a 2D tensor. That's something we're all familiar with in R.

Time series data, or sequence data, is an example of a 3D tensor, even though sometimes that's represented in 2D. You can think of a time series object in R. If you think about a time series, you're actually considering not just the observations, but what happens to the observations over time. So it's really a 3D entity that you're analyzing. It's the features, the time steps, and then groupings of those. Image data is an example of a 4D tensor. Again, you might think of an image as a 3D tensor. You'd say, well, it has height, and width, and depth, which is color channels, like red, green, and blue. That's 3D. But then if you consider samples, again, you end up with a 4D tensor. And similarly, a video would be like a 5D tensor, because it's multiple frames, multiple images. So that's kind of how tensors end up getting represented.

So what is this flow business that we're talking about? TensorFlow programs are not just like an R script that executes. You actually build up a data flow graph. And what happens is those tensors, the data, flow through the graph. And the operations in the graph, and basically their operations in the graph. So an example of an operation inside a TensorFlow graph is like a matrix multiplication, or the addition of a bias term, or taking gradients, or applying some kind of optimizer. So those are like functions that operate on the data, and it's all put together inside a graph. And I'll kind of explain why that's beneficial in a moment.

When you work with TensorFlow, you don't actually program. You can program the graph explicitly. There's interfaces to do that. But typically, you actually write much, much higher level code. And I'm going to explain this R code in detail shortly. But this is an example of a model that I've written in R. And that's, on the right, what it actually looks like as a TensorFlow graph. So the graph, when you're using high level APIs to TensorFlow, is basically generated. And you don't have to reason about it at the level of the graph. But the graph is still there. And why is this graph beneficial?

Well, we can actually think of other examples that we have in R of these sort of intermediate representations. So you can think of like a Shiny application. When you build inputs, and outputs, and reactives, you're actually creating a graph. And then Shiny executes your graph in a way that's kind of optimal and efficient. So the fact that Shiny knows the structure of your program allows it to run your program in a way that's really optimal and straightforward. When you write a dplyr code that works against a database, it generates SQL. SQL is sort of this intermediate language that's then fed to a SQL query optimizer that then executes the query in an efficient way. So similarly, what Google wanted to do with TensorFlow was, if we can get the representation of your model or your program into a graph, then we can run it really fast. We can run it in parallel. We can run it distributed. We can look at the operations in your graph and fuse them together when possible. And then the other thing is we can run this without R or Python, just with C++. So really, the benefits of the graph are all about this idea of portability, performance, and scalability. So this does create some indirection, but I think it has a lot of benefits.

What people use TensorFlow for

So what are people using TensorFlow for? I'm actually going to talk about all of these in turn. They're all listed on our TensorFlow for R website. A lot of people are using it for deep learning, but people are also using it for a lot of classical machine learning. These are some of the examples written up, and these are all long-form blog posts that get into things people are doing. But you can see some text classification stuff, some examples of trying to do predictions on really noisy data. There's computer vision applications, time series applications. And I think there's these examples by actually hopeful and confident the R community, when given access to this library, is going to do some really additional interesting things that surprise us.

I have one example of a project called Greta, which is similar in aims to Bugs or Stan. The idea is to write statistical models and fit them with MCMC. And so the way Greta works is I write a statistical model in R, not in some kind of separate specialized language. And then that model is compiled to TensorFlow, a TensorFlow graph. And that actually uses the R TensorFlow package. And the benefit of this is, A, I'm writing in R instead of a specialized language. But B, because it's using TensorFlow, I can train it on really large data sets. I can train it with GPUs. I can actually deploy the model. So this is an example of something that really has nothing to do with deep learning that people have already done with the R for TensorFlow interface. And I'm hopeful that there'll be more examples of this sort of thing.

Introduction to deep learning

OK, so now let's get into a little bit of what, because deep learning is probably the main thing people do with TensorFlow. And so I think it's important to understand kind of what is it? What is it useful for? What is it not useful for? Should we in the R community, how much should we care about it? And kind of how does it work? So I want to cover some of that ground. And then I'll talk about the R interfaces to TensorFlow.

So at a really, really high level, deep learning is taking some input, transforming it into some output via successive layers of representation. And when I say this really abstract to say input and output, what I mean there is like observations or x data and predictions. So it's x to y, but how do we do that? We do that by taking original input, in this case, a grayscale image of a handwritten number 4. That's our input, and our output is a prediction about what the digit is, the number 4. We have to successfully transform that input until we get close to the output. That's the basic mechanic of how deep learning models work.

So I'm talking about layers. I showed four layers there. What exactly is a layer? And I'm not going to talk about the mechanics of, maybe some of you have seen this before, of neurons and how they talk to other neurons and all that stuff. You can really just think of a layer, and all you really need to consider a layer as, it's a function. It's a data transformation function. It does a geometric transformation of data, and it's parameterized by a set of weights and a bias, just like a linear equation. So you can think of each one of these as just a successive transformation of data. So that's all a deep learning model is, is taking some data and then chaining together a bunch of transformations of the data until we actually get an output, or in this case, a prediction.

So when I say representations, what am I talking about? We really want to take our data and transform it in a way that's closer to the domain of prediction that we're looking for. So this is a really simple example. I've got raw data. I've got a bunch of points, and I want to determine whether a point, be able to predict whether a point is black or white. If I change the coordinate system, all of a sudden, the problem becomes really simple, because everything with x greater than zero is a black point, and everything with x less than zero is a white point. So the idea is we want to transform our data so it's easier and easier to get close to the actual output that we're looking for, the prediction that we're looking for.

If you work with conventional machine learning models, you'll know this as sort of feature engineering, where we're trying to transform our data into a form that works better for the prediction task. And in deep learning, the feature engineering is actually done in the layers. The feature engineering is actually learned rather than hand coded.

The feature engineering is actually learned rather than hand coded.

So back to this example of the handwritten digits, and I'll explain a little bit later how this works. But basically, we're taking raw data, and these layers are trying to filter out irrelevant information, like how the image actually looks. And it's trying to find other things, like where are edges, what are the angle of the edges, and get closer and closer to the domain of output or prediction. So you can think of it as sort of an information distillation pipeline.

And so this gives us an intuition about where the word deep comes from. Why do we call this deep learning? It's not because it gives us deeper insight or deeper models. It's really just talking about the fact that there's multiple layers of representation. A traditional machine learning model might have some feature engineering, and then one or two layers. A deep learning model could have 20 layers, or 30 layers, or 100 layers. And so maybe a more accurate term for deep learning might be like layered representation learning, or hierarchical representation learning. The deep learning maybe implies more than it should, but it's really just talking about the stacking up of layers.

And so what has this method achieved? And you can see from the list here, it's done really, really well on a lot of perceptual tasks, like speech recognition and image classification. And then those, in turn, have been composed together to do things like trying to build autonomous driving systems, to build reinforcement learning systems that play games. So it's achieved a lot, this fairly straightforward mechanism. And to give you an intuition for why it might be able to do that, if you think about a paper ball that's crumpled up, and you think about trying to write a linear equation for how to uncrumple the paper ball, it would be really, really difficult to do that. But if you decompose the un, it's a really complex geometric transformation to uncrumple that whole ball. But if you decompose it into a set of simpler geometric transformations,

i.e. layers, then it's actually straightforward. So a human being might just take that paper ball, and in 30 or so really simple operations, uncrumple it. And that's really how deep learning models work, is that they learn these simple geometric transformations that compose together can do very, very complex transformations, and therefore solve very complex functions, or fit very complex functions.

So why should we care about this? I actually think the domains that deep learning is proven to perform well in are ones that are not often of interest to R users. So these perceptual tasks, like vision and speech recognition, and these reinforcement learning applications, they're not things that most R users are concerned with. So it may be that as an R user, you think it's cool that you can now do state of the art computer vision from R. But it's probably more interesting to say, does deep learning provide improvements on the techniques we have for our traditional domains?

And is there data that we analyze that has very complex sequence or spatial dependencies that's hard to model with traditional modeling techniques? Is there data that requires a huge amount of feature engineering that's potentially brittle to model effectively? So I think it's more interesting to think about, is deep learning applicable, and how, and when, to the things that we traditionally do? And I'll get into some examples of that in a little bit. But I think it's important to note that it's definitely proven to be effective at these perceptual tasks. But it's not yet proven that it's of widespread benefit in other domains, although certainly people are working hard at it.

How deep learning models are trained

So I want to talk a little bit about the mechanism of how these models are trained, what happens, and try to convince you that this is actually a pretty simple and straightforward mechanism. And it's actually kind of surprising that it's able to solve the kind of problems that it solves. So I'll start with a little bit of the basics of how machine learning algorithms work, and then give you an example of kind of how the training loop for a deep learning model works.

So just to define some terms, the way machine learning algorithms typically work is that you start out with a bunch of data. You have x, and you have a known y. And you basically just feed batches of that data into the model. And the model basically incrementally improves its coefficients by examining the prediction, saying how close was the prediction to the actual value, and then adjusting the weights. And the training process is a sort of iterative training process that takes advantage of a loss function to evaluate the model and then tweak it over time. That's kind of how they work.

There's significant kind of differences in orientation between statistical modeling and machine learning. I'm not going to go into a lot of depth about that. I have some links here for you to take a look at. But I think one that you'll notice right away is that oftentimes statistics is focused on explanation and understanding. It's focused on kind of inferring the process by which data is generated. And machine learning really, in a lot of cases, is exclusively concerned with efficacy of prediction. Can we predict things? We don't need to explain or understand the phenomena. We just want to predict it. So as you're going to see, these deep learning models are total black box models. They do nothing to help you explain or understand phenomena. But they work well for prediction. So that's kind of where that orientation comes from.

So let's take a look at a model. This is actually, and again, I showed this before. And I'll get into more detail on this later. Here's the definition of the layers of a model in R. And I'll get into what these are a little bit later. But there's different types of layers. I'm going to compose them together. And I'm going to hope that I can train this model to recognize handwritten digits. And there's an example of kind of what the model is going to learn throughout. It's going to learn these filters that help go from an image into a prediction.

So how does that actually happen? So when a deep learning model starts, it has all these different layers. The layers have weights. The weights, interestingly, are randomly initialized. So in the model at inception, it literally has random weights. So the predictions that it makes are garbage, because the weights are random. And so what happens is that input is fed into the model, and predictions come out that initially are wrong, badly wrong. But what happens is that the predictions are measured against the true known targets with a loss function. So we get this assessment of how good the predictions were. Again, initially, they're quite bad. And what then happens is that loss function is used to update the weights. And that is the job of an optimizer. And again, as I mentioned before, these optimizers can work with just little batches of data, 128 elements at a time. They don't require that all the data is in memory. So we feed the data into the model. We find out how bad the prediction is. We use that to tweak the weights. And then we repeat that thousands and thousands of times until we have a model that performs in a satisfactory way.

So the actual mathematical mechanisms that work here are really straightforward. And the basic mechanics of the whole thing are straightforward. And this tweet is actually from the creator of the Keras library. And he's making the point that there's really nothing complicated going on here. But what happens is that once we take this simple mechanism and we scale it up, it ends up looking like magic.

But what happens is that once we take this simple mechanism and we scale it up, it ends up looking like magic.

And we return to this geometric interpretation that we had earlier. It's basically a mathematical mechanism or machine for taking really complex manifolds of high dimensional data and uncrumpling them.

And the ideas behind this were actually originated 30 years ago. We kind of knew about this. We've known about this for a long, long time. Until about 2012, it actually didn't work very well in practice. And the trick, the thing that changed, is that it turned out that we needed to be able to have really, really large models with lots of big layers and train them on lots and lots of data. And so what happened is that GPUs allowed us to train much, much larger models. And the internet allowed us to collect a lot more data. And that's kind of what changed and made deep learning actually work, even though, again, it sort of was invented conceptually 30 years ago.

So what do I mean by sufficiently large parametric models? This model I just showed you, the grayscale digit recognizer, that model has about 1.2 million coefficients or weights or parameters. And that's actually a pretty straight grayscale digit recognition is a lot easier than a lot of computer vision tasks. So this one has 1.2 million parameters. A bigger model that can recognize all kinds of everyday objects and color images here has about 138 million coefficients or parameters. So when we say sufficiently large, that's what we're talking about. We're talking about potentially tens of millions of parameters that need to be learned. And you can see why these models have very little explanatory or ability to help with your understanding, because you're going to get 138 million parameters, and how do you interpret that? So that's what I'm talking about. And that has made deep learning work very well for computer vision.

Frontiers of deep learning

And what's happening now is that people are beginning to apply it to other areas. And so I mentioned natural language processing, but people are starting to work on deep learning for time series. They're doing it for biomedical applications. They're doing it to create some novel approaches to prediction in some other domains. So I want to talk a little bit about those frontiers and kind of where they stand, and I think that illuminates a little bit of what our posture toward this is in R.

So computer vision is kind of the poster child for what deep learning has accomplished. There's a competition that ran from 2010 to 2017 called the ImageNet Challenge, where machine learning researchers tried to build models that would predict what images are. And the images were like dog, cat, table, chair. So when they started the competition in 2010, no entrants used deep learning, and they had 71.8% prediction accuracy. I believe that was the best model in that year. In 2012, the very first team that used deep learning entered the competition, and that team beat the rest of the field by 11%. And that basically ignited everybody to start looking at deep learning for computer vision. And by the time we got to 2017, in a period of seven years, the accuracy in computer vision went from 71.8% to 97.3%, which is phenomenal. That, more than anything, got people excited about deep learning. And actually, the competition is not going to exist anymore after 2017, because this particular task is basically considered solved. So clearly, it works really well in computer vision and image classification, and you see a lot of applications for it there.

People are also trying to apply it to natural language processing, and it's not clear how good these techniques work. There's a lot of research going on. The plot on the right shows the percentage of papers at computational linguistics conferences that use deep learning, kind of over the same time frame, went from around 35% all the way up to 70%. So again, this could be it's a fad, it's trendy, everyone's trying to see what they can find out. But if you read this paper that I linked to, and these slides are going to be available after the talk, you'll see that people are making progress in natural language processing to do different things. Clearly, a lot of progress has been made in language translation. And I know you may have seen this, Google has a new neural machine translation system that really significantly beats the previous kind of state-of-the-art phrase-based translation, and is approaching human-level language translation. So that's language translation, but there's lots of other things you want to do in natural language processing. Again, people are starting to apply deep learning. There are some interesting early results, but it's not proven that it's going to completely revolutionize that field.

Similarly, people are looking at really complex time series forecasting problems. This is a paper that actually uses convolutional neural networks, which are the same kind of neural networks that are used for images. And so it's looking for kind of spatial dependencies that are invariant across the whole time series. So they're trying to use techniques used for computer vision on time series. And in this case, the data sets they had in this paper, they were able to beat some benchmarks of conventional modeling. But I have not seen in time series. Again, it's not all of a sudden revolutionizing the field, and you should always use deep learning for time series. That's on the contrary. I think there are certain types of time series that it might work a lot better for. But that's really a matter of exploration and research. I have one other time series example of a whole different neural network architecture that's used to, in this case, it's forecasting stock prices. So there's a bunch of articles like this that you'll find. And if you work with a lot of time series data, it might be interesting to read these papers and see where deep learning is performing well and where it's struggling.

There's a lot of work going on in biomedical applications. And this could be everything from patient treatment, patient classification, to actual fundamental biological processes. This paper just came out this month. And it's kind of a roundup of the work that's going on there. And I think they conclude similarly. There's not evidence we're revolutionizing this field or these fields. But there are promising advances on the state of the art worth paying attention to.

This is a very, very interesting example. This paper just got published this month. And it was a study that was done by Google, Stanford, U of Chicago School of Medicine, and the University of California, San Francisco. And they wanted to see if they could provide better predictions about patient outcomes based on electronic health records. And so what they pointed out is that typically, statistical models, electronic health records are really messy. They're uneven. There's different features in different patients. The data is collected in inconsistent ways. They're really messy. So typically, a huge amount of effort is taken to select out a bunch of features, clean up and normalize those features, and then do predictions using conventional machine learning or statistical models. But what they point out in the paper is that when you do that, you're actually discarding a huge amount of data. All the data that you, all the features you excluded because they weren't consistent or coded consistently. Things like the handwritten notes of a physician's, it's all thrown out. So they said, what if we did this? What if we build a deep learning model that considered the entire medical record of every patient, including the physician's notes, and including the fields that are not normalized against each other, and see if we could actually beat conventional methods of doing prediction. And in this case, they did find that they could beat state of the art statistical models in a bunch of categories.

I don't think it was necessarily easy to get this result. If you look at the paper, Jeff Dean is on the paper. He's like the inventor of TensorFlow. He worked on this. So I don't know that anybody could just walk in and get these kind of results. But it's interesting that it's possible. It's a very, very different approach to prediction. It's basically saying that the feature normalization and feature extraction is going to be done by the layers of the deep learning model, not by kind of human reasoning about the data and human composition of like a sensible model. We're just going to learn what the model is. So this is a really interesting paper to read. Again, this just came out in the last, I think, couple weeks.

Problems with deep learning

There are quite a few problems with deep learning models. I've alluded to some of these already. The fact that they're black boxes and you can't interpret them excludes their utility for a whole bunch of things that we want to do with statistics and machine learning and R. They can be brittle. So if you look up adversarial examples, there's a really classic one where you see like two pictures of a panda. And to the human eye, they look completely the same. But one panda has been like tweaked in really subtle ways to force the model to have a totally different prediction. So people are trying to overcome adversarial examples with various techniques. But it's just important. A human being can look at a panda and it just knows there's something way, way, way more complex going on for a human being perceptual task. These models work, but then they're brittle.

They typically need a large amount of data to perform well. Like I said, we've known about deep neural networks for 30 years. And they didn't really work very well until now. So they typically need a lot of data. Although there are ways to transfer knowledge from a model built on larger amounts of data to a model that's trained on smaller amounts of data. So there are ways to use deep learning for smaller data sets. But they typically need a lot of data. And they're very computationally expensive to train as well.

So this creates another problem, which I think is going to get worse and be annoying to all of us, which is that there's a lot of hype about deep learning. There's actually a lot of hype about AI, which as far as I can tell means I have an if-then statement. So it's AI. You know, it's awful. And unfortunately, the current tools for doing deep learning models have become so good that software engineers or even laypeople with no training in modeling or statistics

can actually build these models. And they'll actually think that they've built a model that's really great. They'll say, oh, look, I've got 84% accuracy. I built this model. I know nothing about modeling. I know nothing about statistics. I know nothing about probability. But here's a model. And it's deep. So it's obviously better than whatever anybody else has.

But typically, you can't actually outperform traditional statistical modeling techniques without a lot of effort. So usually these models are going to be bad compared to the models that we build. And so we're going to have to deal with this. We're going to have to patiently explain why, well, maybe that model isn't the best one for us to use. I don't think this is reason for us to throw up our hands and say, we don't want anything to do with this. I think what we need to do is promote a more balanced and knowledgeable and nuanced dialogue about what these things are good for and what they're not good for.

Because in spite of these problems, it does a lot of really useful things. You can actually, this is a slide that Google uses that shows how many model description files, which are basically TensorFlow models, are sitting in repositories all over Google. And you can see, since the release of TensorFlow, that has really accelerated across all areas of the company. So these models, in spite of their problems, their brittleness, their black box nature, can do super useful things. So we want to figure out what useful things they can do for us in R. And that kind of brings me to what we've done in R to let you use TensorFlow.

I think what we need to do is promote a more balanced and knowledgeable and nuanced dialogue about what these things are good for and what they're not good for.

TensorFlow interfaces and R packages

And we actually have a bunch of different interfaces, high level and low level. We have some tools that help you be more productive with your workflow and managing experiments, ways for you to easily use GPUs for training, and then hopefully an adequate amount of learning resources so that you can better understand how to use these tools in your own work.

So top level, we actually have three different APIs for TensorFlow. One is the Keras API, which I'm going to talk quite a bit about. That's a very high level interface for defining neural networks. The R code that I've showed you so far has been from the Keras API. There's another API called the Estimator API, which does more classic classifiers, regressors, support vector machines, random forests, kind of classic statistical machine learning models for TensorFlow. And then the Core API is actually what Greta used. This gives you full access to the entire computational graph. It's really good for library builders, so not something I'm going to talk about.

I'm actually not going to talk about the Estimator API or the Core API much at all. This maps out into a suite of, today, seven different R packages. Three for the interface to TensorFlow. There's a package for working with large data sets called tf.datasets. And then there's packages, as I said, supporting tools for deployment, for managing experiments, and then for interfacing with Google Cloud ML. And I'll talk about those supporting tools in a little bit.

So I've shown you this code before. This is the Keras API, high level API for neural networks. And I will go into a lot more detail about that in a minute. The Estimator API gives you these high level. These almost look like regular R modeling functions, really high level. Say do a regression, do a classification. That's what Estimators is about. And then the Core API is you interacting directly with the graph. Here you can see I'm building the loss function manually, and I'm manually doing the training loop. So I don't recommend people use this API unless you're actually trying to build tools for other people or do some novel, like Greta, do something novel with TensorFlow for R.

The Keras API

So let's talk a little bit about Keras. I'll give you an example and talk about the different components of the Keras API. I wanted to make a little bit of a note about why Keras. And actually, Keras is being, at this point, especially over the last six months, is being promoted by Google as kind of the preferred interface to TensorFlow. Keras works great for end user applications.

You can see this on the left is the Google search for deep learning frameworks over the last three or four years. And you can see that both TensorFlow and Keras are pulling away from other frameworks. And on the right, you can actually see the use citation of deep learning frameworks in research papers. And you can see while TensorFlow has the most citations, Keras, even though it's really great, high level, easy to use API, is also used quite a bit in research. So you can go pretty deep with Keras as well.

Walking through Keras code in R

I want to walk through what Keras code in R looks like. And this is using that MNIST example, again, of handwritten digit recognition. The first thing we do is data preprocessing. And this really just has to do with reshaping and scaling data into tensors. You're almost always going to be dealing with, a lot of times, R matrices. But you're going to get your data off of disk or off of whatever, from a data frame, and you're going to turn it into tensors. So that's a deep preprocessing step.

And then you're going to define your model. So I've shown this before. This is the definition of a model, which basically says, what are my layers, and how are they going to behave? So that's the model definition. And then you're going to compile the model, which basically says, what's the loss function and optimizer that I want to use during training? And what metrics do I want to collect during training? So this is really the heart of making a Keras model. Define the layers and compile it.

I want to make a quick note about this second statement here, model compile. You'll notice I do not assign the result of that expression back to the model. I'm actually modifying the model in place. That's not conventional in R. Typically, in R, objects are by value. You modify the object, and then you return a copy of it that has the modifications. But Keras models are these acyclic graphs of layers. They are stateful. They get updated during training. I haven't shown this yet, but a layer can actually be repeated at multiple places within the model. So these Keras models are by reference objects. And so the code you'll see when we mutate them, when we compile them or fit them, it's all done in place. I just wanted to make that clear, because I think when you see this code, it's different from maybe other R code that you've seen.

So now training. The training process is done through the fit function. And we basically say, here's our data. And in this case, we're saying we're going to do 128 samples at a time to the model. We might have millions of samples. We'll just do 120 at a time. We want to traverse the input data set 10 times. That's what epochs is. Again, since we're going to iteratively learn on batches, it's not enough to see the data once. We're going to see it multiple times. And then as we're fitting, we're going to hold out 20% of the data and use that to validate that we're not just overfitting to our data set. We're not basically a deep learning model can actually just memorize the data. And so it's giving you a function that really isn't useful because all it's done is memorize the data set. So we hold out data. And we test it against that to make sure we're not just merely memorizing data.

So that's what training looks like. When we're done with training, you can see we assign back to a history object. And we can plot the history and see how the training proceeded and how our accuracy and loss improved or didn't improve during training. Later, we can evaluate the model, which is taking yet another set of data that we've held out, making sure the model has never seen this data. And then we test to see how good the predictions are. Was the model really an overfit to the data? Or does it generalize well? And then we can use predict or predict classes to actually generate predictions from the model.

So I actually want to show you a little bit of what that looks like in RStudio. This is a complete end-to-end script that I just showed you in slides. So again, when we start, we'll load the data. The MNIST data set's actually built into Keras. So I get the raw data in. And then I'm going to turn that data into some matrices that I'm going to be able to feed into the model. You can see on the side now I've reshaped the data into a bunch of matrices. And then I define the model. And then I compile the model. I can print a summary of the model to see how many parameters I have, what the layers are, and how they work. And finally, I fit the model.

And as the model's being fit, we're going to actually show you the training metrics in real time in RStudio. And this is actually extremely useful because a lot of times when you fit a model, you're overfitting, your accuracy is not improving, the model doesn't work. And it's really good to be able to visualize that as you fit it so then you can stop training and say, nope, that architecture doesn't work, those hyperparameters don't work, I need to try to do something else.

I can also plot the history. And you can actually get at the data frame behind that if you want to do various types of custom visualizations of your training history. Evaluate to see what my accuracy looks like on data I haven't seen before, and then generate predictions. So that's just the basic mechanic. Models can get more complicated, but some Keras scripts are really this simple. This is all you do. Not to say that the task of actually designing and building and getting the model to work is simple. A lot of times, that takes a huge amount of iteration.

Keras layers, compilation, and the cheat sheet

So layers is kind of the core construct in Keras. And there's actually 65 different layers available, and you can create your own layers. This is just a sampling of some of them. I'm going to talk categorically about some of the types of layers that there are.

There's dense layers, which are kind of the staple of neural networks. These work the same way that traditional neural networks that were written about and used in the 70s work. And they're kind of a staple of every deep learning model. Usually, if you're using more sophisticated layers at the end of the model, you'll have these dense layers. And these dense layers are really just a bunch of weights and biases that are applied and cascade through other dense layers.

Convolutional layers are used for computer vision, most commonly. And they're basically trying to find spatial features. And they're trying to find spatial features that, when they learn, like, for example, what an edge is or what an eyeball is, that knowledge is actually transferable to other parts of the image. So if I see an eye in a certain quadrant of the image, and then I see later on an eye in another quadrant, I can still recognize that as an eye. So the reason it's convolutional is we're building up these filters that recognize patterns. And then we're convolving, sliding the filter across the entire image. And you can see an example of that here. So again, there's much better explanations for how this works. But you can think of convolutional layers as really good for image processing, spatial dependencies, that sort of thing.

Recurrent layers are basically layers that have some memory or state. And so if you're dealing with sequence data, like time series data or text, where it matters not only what I'm seeing now, but what I've seen in the past, recurrent layers are able to maintain state. So they're oftentimes used in these sequence-oriented applications.

And then embedding layers, for those of you who do natural language processing, is a way of vectorizing text. So one way to vectorize text would be to say, I've got an instance of the word cat, an instance of the word dog, and they're just classes. It's just cat versus dog. But another way would be, I've got a vector that represents semantic relationships between words. So in one of the dimensions of that, one of the axes of that vector, I've got a thing that says, oh, cat and dog are both animals. So embedding layers do this kind of vectorization. And you can actually learn the embedding jointly with building the model, or you can load a pre-trained word embedding that we conventionally use, like word2vec. You can load one of those embeddings into an embedding layer. So there's just a few of the layer types, but again, there's 65 layers. And a lot of what doing deep learning is is trying to figure out what is the right composition of those layers and behavior of those layers to get a model that works well.

So in Keras, once we define our layers, we compile the model, and what is that really? The compilation of a model is really converting the layers into a TensorFlow graph, and then applying the loss function and optimizer to those. That's what compile, the compile step really does.

There are a wide variety of loss functions available in Keras, and you can actually write your own loss functions. So depending on what you're doing, you'll use maybe a custom loss or maybe one of the built-in losses or more than one of the built-in losses. There are many optimizers that are built in. Again, these optimizers don't require that all the memory is in RAM. There are many metrics built in, and you can write your own metrics. So these are ways of assessing how the model is training, how it's behaving, how well it's doing.

All of this is summarized in a cheat sheet that we have. Don't worry about copying this URL down because I'm going to share the link to these slides, and this is available from the TensorFlow for our website, but this cheat sheet kind of goes through a lot of the same content that I just went through, but I think it actually is a much more clear treatment. It's really excellent.

Deep learning examples with Keras in R

So I wanted to go on and talk about some examples that we've written up of using the R Keras API to do deep learning. These are all available on this gallery page of our TensorFlow for our website, and each one of these is a blog post that writes up in detail kind of the motivation, what we did, what worked, what didn't work, intermediate data visualizations. So a really good way of exploring these different use cases.

The first one actually demonstrates how to do what I mentioned earlier, transfer learning, where you, here we're basically going to try to predict whether an image is a cat or a dog, and we're only gonna train on 2,000 input images, and the way that's gonna work is that we're gonna use a model that was already built, that knows about edges and things related to identifying things like cats and dogs, and we're gonna transfer the knowledge of that model into our new model. So this is an example of using deep learning with a smaller data set.

There's another example that talks about time series forecasting of weather, temperature data, and it talks about different techniques for different things you can try to make a time series model perform better, including stacking up multiple recurrent layers together by directional recurrent layers, which look at both temporal orders of data. So this is a good kind of introductory primer to doing time series.

There's a really cool example we have where we're trying to, for cancer immunotherapy, classify peptides to figure out how they're gonna bind, if they're reintroduced, and this is using deep learning. We show the same classification task with a random forest and with a simple feed-forward neural network, and it's a really good introduction to how you might use deep learning with this sort of data.

Another example is looking for credit card fraud, and what's interesting about this is that you've got, I think, something like half a million transactions, and only 2,000 of the transactions are fraudulent. So this is a way of trying to find signal in those 2,000 examples of fraudulent transactions and be able to still get good predictive accuracy,

even though we don't have that much data to train on. So that's what that is.

This is a text classification example, which says we've got questions from Quora. Are they duplicates or not? It's based on a Kaggle dataset. One of the interesting things about this, we also show how to create a shiny application that's a front-end for this model.

It's actually quite straightforward to do that, and this gives you a sort of simple introduction to doing that.

Another example is using deep learning to predict customer churn, and this also demonstrates a couple other interesting things, one of which is the LIME package, which I'll talk about in a minute, which is a way of, it's not a way to understand how the model works, but it's a way to understand for an individual prediction what features contributed to that prediction. So that LIME package is demonstrated here, and we also demonstrate, again, building a shiny application on top of a deep learning model.

Another example is learning word embeddings from Amazon Reviews. So this is, we've got, I think these are the Fine Food Reviews dataset from Amazon, and we're trying to figure out what words are semantically similar. So this is just a plot of some of the, it ends up being a multidimensional embedding, but this is an example of doing word embeddings.

Explainability of predictions

Couple examples about addressing explainability, because again, while 138 million parameter model, looking at it doesn't tell you anything about the phenomena, it is actually possible oftentimes to understand why a given prediction was made by a neural network. Here's an example of, we're classifying images, and we've predicted that this image is an African elephant, and actually by inspecting the gradients of the intermediate layers, it's possible to pull out a heat map that says, these are the parts of the image that most caused us to do this classification.

So then we juxtapose the heat map over the original image, and we actually see, this is why the model, these are the things that contributed to the model saying that this was an African elephant. And then similarly LIME, which allows you to take an individual prediction and say what features contributed to that prediction.

So there is work going on in explainability of predictions, it's not the same thing as I look at the model and it explains the phenomena, but a lot of these models are gonna be used if they're being used for things like patient care or making recommendations about surgical procedures or approving or denying loans, they have to be explainable. We have to be able to say, we have a machine learning algorithm, but this is why the machine learning algorithm is coming out the way it is.

Deep learning frontiers and experimentation

So kind of coming back to this idea of frontiers, many fields have a deep learning frontier that is not yet reached or even well approached, and in most of these cases, the traditional statistical modeling and machine learning methods are cheaper and more accurate. So it's really pretty, once you get out of computer vision and language translation, a lot of this stuff is in the research stage, it's in the speculative stage.

And I think there's different approaches you could take to that. Right now, creating an image classifier that has a super high accuracy is just completely trivial. So that field, that frontier is explored and understood and applicable.

But in other cases, the frontier is far away. So you could wait and see, wait till, okay, I do this type of time series forecasting, it's not proven that deep learning helps me, I'll just wait and see until it is proven and then I'll apply it then. Another approach could be to try to see what contemporary research is doing, see both formal research and what people are publishing on the web, and evaluate that and try that with your data and your work and see if it improves things. And yet another approach is to actually try to, in your field of endeavor, try to approach the frontier, try to see if you can find novel applications for deep neural nets.

So all of this requires, it's great to show examples that do interesting things, but it does require a lot of work and time and patience and iteration to get these models to work well. So in practice, there's a huge amount of experimentation that's done with model architecture, with hyperparameters. And what happens is people gain better intuition about what sorts of models and layers and other things work well for the class of data problems that they have.

So it tends to require a lot of iteration, it tends to require a lot of compute power, and it sort of tends to require some related tools to help you kind of streamline your workflow and make the most of the compute power you have. And that's kind of the next piece that I want to talk about is the tools that we've built to help you with the process. Kind of first and foremost is using GPUs.

Using GPUs and cloud resources

Some deep learning models work great on CPUs. Ones that use convolution or recurrent neural networks tend to be very slow on CPUs. So if you go to our TensorFlow for R website, there's a bunch of resources about different ways you can use GPUs. One is if you have a high-end NVIDIA graphics card, you can use a GPU on your local system. There's cloud services that let you do batch jobs that use GPUs. You can set up a cloud server with RStudio Server on Google Compute or Amazon or Azure. And there's even ways to actually have a virtual cloud Linux desktop that has a GPU. So all that's covered on our GPU page. I won't talk about that further, but it's definitely something to look into if you want to explore.

The tfruns package

We have another package called tfruns, stands for TensorFlow runs. And the idea here is that because a huge amount of experimentation is required with deep learning models, you're going to run a ton of training jobs. Instead of using the source function to run those jobs, you use a function called training run. And what training run does is it records everything about that job. The source code, the output, the performance of the model, the hyperparameters you use for the model. And then you basically have a data frame at the end for all the model runs that you've ever done. And then you can compute on that data frame to try to understand better what's working well and not working well.

It also gives you reports on each runs that summarize, again, the code, the model, the metrics, the performance of the model. So every single time you do a training run, one of these reports is generated. And then you can compare runs. So if you find, okay, I've examined my data frame. I found that these are my best two performing models. What was different about them? And you can see in this case, it's showing a source code diff saying I changed two things in the source code. And that accounts for that difference in performance.

And there's another piece called flags, which basically says if there's important aspects of my model, it's good to externalize those, get them out of my source code so that I can kind of pass these flags to my script as a parameter. So you can see I'm defining a set of flags here. And then I'm using the flags when I define a layer. And then when I train the model, I'm passing one of the flags. They're basically kind of command line parameters for your script that let you easily vary things.

And then that in turn enables this construct called a tuning run that says, okay, I've got two types of dropout. In this case, I'm gonna vary dropout from 0.2 to 0.4. And I want you to try every combination of those two hyperparameters and then tell me what the best model is. And if the number of combinations gets very, very large, you know, into the thousands, then you can actually sample from that and only run 50 training jobs rather than thousands of them to try to evaluate what the best model is. But these sort of tools end up being critical to getting deep learning models that perform well because you have to try so many different things.

Cloud ML for training and hyperparameter tuning

The Cloud ML package is our interface to Google Cloud ML. And Google Cloud ML basically lets you train models, sort of, you sort of submit batch jobs to Cloud ML and then the model trains. And what's really cool about it is that they have GPUs and they actually have some really, really fast GPUs and even machines that have multiple fast GPUs available. And they also have a service for doing this hyperparameter tuning that I just talked about. So the idea is you want to work from the comfort of your laptop with smaller down-sampled data. And then when you want to actually do your training, that's going to take three hours. You submit it to Cloud ML and then it takes 20 minutes, hopefully, instead of three hours.

Similar to Training Run, there's a function called Cloud ML Train. And when you give an R script to Cloud ML Train, it'll actually upload the contents of the working directory to Cloud ML. It'll actually figure out what packages you're using and make sure those packages are installed on the Cloud ML server. And then when I do this, I can actually specify the kind of hardware I want to use. So in this case, I'm saying use a standard GPU. In this case, I'm saying use like super pumped up, latest and greatest NVIDIA Tesla GPU.

And then you kind of similarly, once you do a training job, you can collect the results. You can get all the files that were generated. You can get a report about how it did. You can see all your jobs in a data frame.

And then once, if you're using these flags that I talked about, you can actually do hyper parameter tuning. So in this case, I've defined my flags and I've used them in kind of composing my model. And I can write a YAML file that I give to Cloud ML and I can say, here's the parameters. Here's how I want to vary them. And here's the metric that I'm trying to optimize for. In this case, it's validation accuracy. And here's how many trials I want to run. So this is a way to search over huge spaces of models. And it's all done remotely in the cloud. And you're kind of just only charged for the compute that you use in the execution of those jobs.

So when the job is done, you get a summary of all the jobs, what the flags were, and what the accuracy was. And so you can figure out what architecture and hyper parameters work best for your model.

Deploying TensorFlow models

I talked a little bit earlier about how TensorFlow models can be deployed without an R runtime dependency. So the tf.deploy package has tools that facilitate doing that. I won't get into all the, there's actually another talk about this in the interop session one that will go into more detail about this. But the idea is you get a model, you export it, and then you can, for example, serve that model using a REST HTTP API.

You can serve it using an open source project that Google has called TensorFlow Serving. You can serve it from RStudio Connect. You can serve it from Cloud ML. So these models, the nice thing is these models can be served from a whole wide variety of different sources. And again, there's no R runtime dependency for serving the model. It's very easy to embed these models inside Shiny applications. Couple examples of that on the gallery.

And then what's really cool is you can actually deploy these models not just to servers, but you can deploy them to embedded devices. You can deploy them to mobile phones. You can actually deploy them into the browser. There's a library called Keras.js and a library called DeepLearn.js that lets you take TensorFlow and Keras models and run them directly in JavaScript. So this idea that a model is this graph that's independent of the programming language used to create it is a really, really powerful one, and I think, hopefully, one that we can take a great advantage of in R. Again, there's a talk talking about deployment in the interop session.

So this idea that a model is this graph that's independent of the programming language used to create it is a really, really powerful one, and I think, hopefully, one that we can take a great advantage of in R.

Key takeaways

So what are the big, big takeaways that I hope you internalize a little bit from this talk? One is think of TensorFlow not just as a library that people use for deep learning. It is a general-purpose computing library, and it has lots to offer us, and we should be exploring those things like what has been done with the Greta project.

Deep learning has made a lot of great progress in some fields, and a lot of people are working hard at making progress in other fields, and it's likely to increase in importance in those fields. It's hard to say exactly how much, but I think it's something that a lot of people are working on. It's gonna be relevant to our work, and I think, most importantly, the work that we've done for the TensorFlow R interfaces, we have a great set of high-level APIs, different levels, we have high-level and low-level, so doing deep learning in R is now really easy, and you have the state-of-the-art tools, you have the tools that are used by researchers and practitioners, the best tools used by researchers and practitioners today, and we have low-level interfaces for doing more innovative things.

There are a couple more talks in the Interop track at 11. There's a talk from Michael Quinn, who's from Google, and he's gonna talk about modeling predictions on Google Store data, using the tools that we talked about today, and then we have this deployment talk that comes immediately after that, so check those out if you're interested. There's a website that we have, tensorflow.rstudio.com, that really has a lot of the content that I covered in this talk, obviously much deeper. That website covers everything you need to know about TensorFlow and R,

and this is actually really important. If you wanna get into this, there are two books that I highly recommend that you read. They're very different books. One is this book, Deep Learning with R, that just came out a couple days ago, and my name's on the cover, but I didn't write any of the prose in the book. It's written by Francois, who's the gentleman who created Keras, and I took his book, his Keras book, which covers the Python interface, and I worked with him to adapt it to R. So this is a book that really covers the concepts behind deep learning. Mostly it's a conceptual book, but all of the examples are done in R, using the Keras interface.

There's actually no math in this book at all. It's all of the mathematical intuition is provided by showing you R code. So that's one book. Great conceptual introduction, all code, no math. The other book is kind of the most important book, I think, right now in the field, sort of providing the conceptual foundation for deep learning. This book has no code at all. It's all math. So depending on how you learn, and how you like to approach things, read one or the other first. But these, they're very different books, but I think they're both essential.

So thank you very much. These slides are available at the URL displayed here. And the other thing I would urge you to do is if you're interested in this stuff, on our TensorFlow for R blog, we're posting new examples and case studies all the time. So if you wanna kind of keep up with the work that we're doing, and other people are doing, subscribing to that blog is a great way to do that. So thanks very much, and I'd love to hear what questions people have. Thank you.