Resources
Oct 21, 2023 |
1h 21min
|
325 views
|

Charla Plenaria: Max Kuhn

Introduction to Tidymodels

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

And now I'm going to switch to English because our next speaker speaks English so I'm going to switch. The next keynote will be Max Kuhn. He works as a software engineer at Post-it. He leads the tidymodels team.

And on 2020, when we were supposedly organizing the event here we decided to invite him but the pandemic and stuff so we have to postpone the invitation and at that moment when we decided to invite Max, we looked at our list of cool people that do awesome things and he was there so let's invite Max. And on 2023 when we were absolutely sure that we were going to hold the event here we invited him again so let's come to Montevideo and he didn't answer our email so we moved him from the cool people list to the black list for a couple of weeks until we discovered that he actually answered our email but it went to the spam folder.

And we solved the issue and now Max is here to talk about the tidymodels universe so thank you Max for coming.

You can hear me okay? Excellent. Thanks for sticking around to the end of the conference, I appreciate it. Now I've been using, you may have noticed this, we've been using Google Translate during the sessions and I've realized if you're going to do that and translate it to Spanish or Portuguese, I will speak try to speak slowly and clearly.

I'm saying this because, let's see, my favorites so far have been so I wrote a package called Carrot and has anybody in the US if somebody calls you a Karen has anybody ever heard that before? It's kind of like a really mean thing to say and it translated Max Kuhn's Karen package.

So that was that and then ggplot got translated to was it like jpplop2 yeah so that's what we're going to call it from now on we're going to send it to Cray and break all your dependencies it's a new name.

Introduction to tidymodels

So I'm here to talk about tidymodels, we did a tutorial on this, hopefully it won't be too bad if you were there and hopefully it will be good for people who weren't. So if you want the slides it's at the bottom of the slides, there's a little bigger font here if you want to see it, my github handle is topepo so it's 2023 Latin R.

So at first I'm going to tell you why maybe tidymodels and actually Carrot too exist. R is very very very good at many types of modeling, it's always been stellar at things like mixed models and hierarchical models, survival analysis, and there are just so many things that it does in terms of modeling.

For a long time it's had quite a long history of having really good predictive modeling and machine learning models, so there's a lot of really good things if you're doing modeling in R. You know the formula interface if you ever use that, like y tilde x, that's been around since I think it was like 1982 believe it or not in the S language, and so all the things they did back then in the S language have really carried through R and so we've always benefited from having really expressive ways to do modeling.

And it's also pretty much had cutting edge models, so things like random forest the first real implementation of that that people could use that wasn't Fortran was built in R and so a lot of times people who do modeling they might use R as their first sort of like prototyping language. So outside of like deep learning and some other things especially as tree based models and regularized models I think you see things in R before you see them anywhere else.

And if they're not there the last really good thing that's pointed out here is R and the S language originally was built as an interface to Fortran wrappers if you can believe that, so for quite a while in the DNA of R and S it's always been really easy to take things that R doesn't do well and do them much faster or better in C or Java or Python or whatever it is and so there's quite a lot there to love and use and it can be very effective with it.

The consistency problem

But there is a downside and it's mostly about consistency and so you know some examples that I'll go into a little bit more detail come down to things where I think package managers and I'll include myself in here early on, I think when we write packages we kind of write them for ourselves usually and so a lot of times the scope of the package or the things it can do or the things that it doesn't do a lot of times it's largely informed by the things that are important to us at the moment right so who we work with.

You know one said you know I really wanted to eventually write a package that I love that nobody knows about right and so what happens sometimes is somebody writes an interesting modeling package but there's a lot of things that other people might want to do that maybe they don't know about or haven't thought about and it doesn't really get implemented in the package and that can make it kind of difficult to use sometimes.

So just some examples of the consistency problem. Let's say you're fitting a classification model and let's say you have like two classes like yes or no and you want to get the class probabilities back, you might start off by fitting like an old school like linear discriminant model and the standard way of doing that is to take that model fit and put it into the predict function and if you use that particular model and you want to get the probabilities back you don't really have to do anything you can just use the predict function and give it your object.

But for a lot of other models you have to say something like type equal and this is just this slide is usually a lot longer with all the options you have but you know there's type equal response for logistic regression there's sometimes it's prob which is the one I like, sometimes it's posterior, sometimes it's probability so you know when I was doing this for a living like in before 2000 I'm old, you know this is very very frustrating because you have to remember all the minutiae of every particular package that you would like to try and that was kind of a pain.

You know this is very very frustrating because you have to remember all the minutiae of every particular package that you would like to try and that was kind of a pain.

The other thing is R generally has two interfaces the models there's the formula method that I discussed and then there's also what we just really call like the non formula method and that's where you have predictors in some matrix or data frame x in your outcome in some vector or matrix or whatever y and then you'd say the function and give it x and y and sometimes you want one or the other or sometimes you might want both depending on how your data come in and things like that.

And so here are five fairly popular machine learning or modeling packages. Let's pick glimnet, so between the formula method and the non formula method raise your hand if you think it has the formula method. Okay thank you raise your hand if you think it has the non formula method. Okay wow, okay one versus three thank you.

Hannah well here's how this all shakes out so if you're using glimnet it actually only takes matrices which is kind of a pain because if you have any categorical predictors it's your job to turn that into indicator variables if you want to transform a variable that's your problem, the formula method kind of does that automatically for you and then you have to when you make new predictions give it a matrix and it's very limiting, so if your data is not just a bunch of numbers already that you know you do the work.

So you know we get around that but it is what it is. Rangers are really good random forest package, it's pretty fast and it does a lot of things and it has both the formula method and the non formula method which actually is not correct, you can use the formula or use x and y the problem is it's all in one function so they have a function that have arguments formula and data and x and y which is kind of weird but we can get there.

Our part is a decision tree package, very old it has just the formula method. I long ago put in an issue to get the non formula package or non formula interface and the maintainer sent me the sources and says you should maintain it then I am not maintaining it. And then survival is for survival package, it is also formula only and then XGBoost which a lot of people love, not only does it not have a formula method you have to convert the data to a special sparse matrix format, you append your outcome to that and if it is a classification model you have to convert your outcome to zero based integers. So this just gives you a sense that these are all used quite a bit, all these packages, so sometimes you are doing a lot of work for these packages so this isn't really something that we like people to deal with.

The role of a systems statistician

So I used to work with engineers a lot and some of the sharpest and most functional engineers I worked with were called systems engineers. So like you think about your phone and there is a whole engineering crew at Apple or the people that make Android phones that are battery engineers right and they have engineers that work on the screen to optimize that and the antenna and they have all these groups of engineers and what a systems engineer does is make sure all that stuff works together, that one thing is not taking too much power so the other thing can't work and so on. They just make it all happen not at the end but along the way.

And I feel like I realized maybe like a week or two ago that my professional life mostly has been spent as what I would think of as a systems statistician because we have a lot of packages that do really interesting things and we want to use them but they are not necessarily made to be consumed to be used somewhere else or in a different context than the original author intended. So that is a lot of what we do, that is a lot of what I've been doing, and that is absolutely the goal of tidymodels.

As what I would think of as a systems statistician because we have a lot of packages that do really interesting things and we want to use them but they are not necessarily made to be consumed to be used somewhere else or in a different context than the original author intended.

So our job is to really make our less frustrating but really it's really not that bad, it's very functional and with a few extra tools you can get very very far and build very effective models. And I stay here like we usually talk about predictive models and machine learning, tidymodels can do pretty much everything, we have packages for hierarchical models now survival models and so on so it's not just like a machine learning library.

So you've probably heard of the tidyverse, I know that's kind of a ridiculous thing to say but I've done this presentation where people looked at me funny, and the tidyverse is built to have packages that work really well together to have a common sort of design goal, the naming is very similar and it's very very thought out in how you interface with the software.

and anything else I've ever seen and when Hadley first started working on that he had these goals that are listed here don't make a new data structure that somebody has to learn try to stick with data frames if you can probably the most important one here is design for humans can anybody, raise your hand if you know the function name if you have a matrix and you want to invert it, does anybody know what the function name is?

you first solve there's no inverse function that's not designing for humans it's those kind of things, we want to make things obvious for you, we don't want you to get a result back and be surprised wait, why is it in a list and not in a data frame that kind of thing, so when we think about designing for humans in terms of modeling, it's a little bit different from what they do in the tidyverse but at the end of the day we want you to enjoy programming and not be constantly frustrated by it

in some of the things that we do is we try to name things and optimize them for auto complete, so if you're doing model tuning, all of our function names that do that start with tune underbar and then if you're in RStudio or VSCode you can just tab and then you get the whole list of things, whether they're grids or tune objects or things you can extract, like we try to make function names as consistent as possible Hannah and when I don't they fix it

so the other idea is this notion of tidy data there's a link here to when I have these papers that really every variable should be in a column, every row or every observation forms a row, that's a little bit weird for us sometimes because sometimes we have records that have multiple sub-records so if you're a patient in a clinical trial or you're being followed longitudinally the patient is really the observation, they're like the independent experimental unit and they might have multiple rows in the data set so some of this we have to deal with a little bit but our idea is we want to create modeling packages that work well together that do a lot of things and that you know when you start using them, even if you haven't used the other parts of it, you'll already know what to expect when you get there

so I've been thinking about modeling for a long time and if you look at the tutorial I mentioned that in the process of having a career doing modeling before coming to our deposit you can edit that out, right? There were mistakes were made you learn a lot along the way and so a lot of that informs how we design our software

Design principles of tidymodels

so the first bullet point there is type equal prop, that's just remember that one, right? So when you use our code we'll make the translation for you for whether it's probability or posterior or whatever if you want to use the form method or not the form method we'll handle that for you so we don't want you to have to worry about that. Another thing that's important to me over the year is resampling and empirical validation that when you have a model and you want to know if you want to get something out of it you should be able to quantify its performance hopefully using something like cross-validation of the bootstrap. I've sat in a lot of presentations where people present results with logistic regression and how important it is that this p-value is for the interaction is less than .05 and then you ask what the accuracy of the model is and it's like 60% and you're like, okay so there's a lot more to qualifying and validating your model beyond just doing predictive modeling

Another thing that we do in tidymodels is it's designed with some hidden guardrails at least in machine learning a lot of the problems that occur that you don't necessarily know happen until you get new data those are the result of people using maybe the wrong data at the wrong time for the wrong thing or using it for multiple things and so we sort of, you sort of have to go out of your way in tidymodels to sort of use the data inappropriately by accident

it's designed with some hidden guardrails at least in machine learning a lot of the problems that occur that you don't necessarily know happen until you get new data those are the result of people using maybe the wrong data at the wrong time for the wrong thing or using it for multiple things and so we sort of, you sort of have to go out of your way in tidymodels to sort of use the data inappropriately by accident

and so we want people to also be able to use the infrastructure to make your own packages pretty easily and we want to encourage people to do things that they necessarily couldn't because I've heard a lot of people say yeah, that is probably a better way to do this but I don't know how to use that software or I don't have a license for that or it's not been written and I'm probably not going to write it. So we want to also sort of like the next slide has the phrase leveling up. Does everyone know what that means? It means like how can I improve me or my tool set to get more out of what I'm doing

and just some examples of that is a lot of times we have variables like predictors in our data and we use a model that can't handle them as character data, you know, because they're categories and so we have to convert them to indicator variables there are a lot of different ways you can do that and a lot of times there haven't been there have been methodologies out there that people just didn't have available to them or the package wasn't easy to use and so on. So just as examples, there are things called effect encoding methods that can be very effective in certain situations feature hashing is an excellent tool if you have like categories that have maybe like thousands or millions of levels and things like that. Also multiple choice predictors. I've had codes laying around in scripts for years that take, you know, if I had a model and a predictor was like what languages do you speak, you know, some person might have one, somebody might have four and so having like a way to deal with multiple categories is really helpful. So we're trying to institutionalize as much of that as we can in tidymodels so you can just use it and not have to carry around your little bits that you've been doing.

I had these slides already written before I saw Di's presentation but also some modern dimension reduction methods like UMAP which he showed and these fancy name things like manifold based multi-dimensional scaling and things like that. So, you know there are things we see in papers and all that but we don't necessarily have the tools to fit them and so we want you to try to use them. Maybe they're not effective maybe they are but having the ability to even try that is important to us. You know there's other things like a lot of different imputation tools are out there so we just want you to be able to use this stuff. So, you know, if there's something like yeah I use this all the time but I can't find it anywhere just come to our repos and put an issue we'd like to hear from you.

Learning resources

Alright, so if you want to learn more besides this there's three main resources. There's tidymodels.org. We spent a lot of time on that during the very beginning of the pandemic just sort of optimizing that putting a lot of articles in there searchable list of functions a lot of really nice sequential set of getting started articles. Julius Hilge and I wrote a book on tidymodels. You can get it from O'Reilly or you can just go to this link and take a look at it. That's very helpful if you want longer form documentation. And also some of you may know after the other day we keep all of our training materials in the same place. We keep all the old versions so if we change data sets you can always get to the version that you used but you can go to workshops.tidymodels.org and there's a lot of training materials there. And all this is licensed in a way that you can use it in your classes or for whatever you need.

Looking at some code

Okay, so let's look at some code maybe. Like the tidyverse we have like a meta package. So if you load tidymodels you see this big amount of output you get back. We load things like dplyr and ggplot which are not part of tidymodels but you're probably going to want to use those. And also like tidyverse we list these conflicts. You know things like dplyr has a filter function but the stats package has a filter function. If you just type filter the one that you get is based on which package was loaded more recently. And so one thing that we added here is this thing at the very bottom called tidymodels prefer and what that does is it has uses the package that Hadley wrote called conflicted that manages these conflicts. So after running this function assuming you want to use dplyr and tidymodels it will sort of default your filter no matter how it gets loaded to use the dplyr one. So you don't have to use this but if you're sort of tired of dealing with these and have the wrong function from the wrong package this can be a big help for you to resolve that.

Example data: restaurant deliveries

Alright so I have some example data. It's a little bit more than 10,000 data points and it's something about restaurant deliveries like food deliveries. So basically we have a time period from when the order was put in to when it was delivered. There's a few variables. There's the order time is like the hour. You know 0 to like 24. There's a day of the week. So it's a factor. It's Monday, Tuesday, Wednesday that kind of thing. There's distance and also there's 27 pre-made sort of indicators and this is like the result of like a multiple choice thing. These are counts of the things that were ordered so like item 3 might be somebody ordered a nice tea or something like that. So we want to take this information and try to predict how long it would take you or me to get our food at home or wherever it's being delivered to. So it's a fairly decent sized data. It's 10,000 data points. There's not that many variables.

The first thing we'll do is we'll look at the outcome. I should say if anybody here does survival analysis you're probably wondering if any of this data is censored. This is all complete data. Censored means if I put an order in and it's five minutes after I don't know when it's going to get here so I don't know the actual delivery time but I know it's at least five minutes. And so these are all complete values where they've actually had the delivery and they know what the time was. And you can see it's right skewed which is pretty common with time data. It's non-zero. It ranges from maybe like 12 to around 60. It's right skewed with a little bit of a mode here.

So one of the things we might think about is do we do anything special about this? One thing you might think about doing is logging the outcome and then doing that. I'm just going to keep the outcome the way it is. And the only other thing I'm going to do with it special is I'm going to split the data up. I'm going to split it into a training set, a test set, and a validation set. And the validation set is sort of like a test set but you can use it along the way to figure out how well your model is using or how well it's working. The test set is really at the very, very end when you want to just validate what your results were. So when I do that split, what I don't want to happen is I don't want to have, you know, all of these sort of like outlying extreme data points end up accidentally in the test or the training set. So when I use this function in tidymodels called initial validation split, it'll put 60% in the training set, 20% in validation, and the remaining 20% in testing. And what it'll do is it'll do that three-way split by chopping this distribution up into quartiles and doing the split in each quartile of the data and then recombining them together. So the distribution of this outcome and all three of those data sets should be very, very similar to one another. So we can make sure by doing that, we can make sure the distribution here looks the same across the training set, the test set, and the validation set.

So in the code here, you know, we get to split about 6,000 data points for model fitting, about 2,000 for validation, and about 2,000 for testing. We try to make our function names as obvious as possible. So if you have the object that has the information about the splitting, you want to get the training set out, we're going to name that training for you. And validation is the function that brings back the validation set. So, you know, this takes a split and you get a data frame out of here with the appropriate 6,004 rows and so on. Now a lot of times I talk about resampling, like cross-validation, the bootstrap, and those are ways to use your training set to get good estimates of performance that are not overly optimistic. And I'm not going to be doing cross-validation or resampling or the bootstrap or something like that. I'm going to use my validation set sort of as like a single resample. And to put that sort of into the tidymodels format the same way you would get the same format as cross-validation, do a little extra step here and take that split object and pull out the validation set in a way that we can consume this by some of the functions we'll use later.

Exploring the data

So, you know, I think I see this on one of the slides upcoming is one of my college professors said that the only way to be comfortable with your data is to never look at it. And so, you know, when you look at this data a little bit, you look at the time of the order, and clearly the delivery time increases with that and it's maybe not linear. So this is something, if we're going to fit like a linear regression here, we need to think about that there's a non-linear effect here. I use geomSmooth, and what geomSmooth does under the hood is that you use something called a spline function. And what a spline does is it takes the original time variable and sort of expands it to these kind of weird extra columns, and those extra columns are the ones that are used in the linear regression. So this smooth line here that you see here is the prediction from like a linear model that uses maybe five or ten of these extra non-linear terms based on the order time. That'll be relevant in a minute, so that's why I'm lecturing on geomSmooth. But this is something we need to think about if we're going to fit like a linear model.

But if you look at it a little more closely, it's actually a lot more complicated because it turns out if you look at that same trend and estimate it differently for the day of the week the order was made, it's quite different, that Friday and Saturday have a very, very steep non-linear curve, whereas something like Monday is a lot more blunted. So if we just did this and put it in the model, we're going to underfit because it actually depends. And that's what a statistical interaction is, is when you have two predictors and their relationship with the outcome, they depend on one another. So in this case, just knowing the day of the week is not enough, and it's also not good enough to just do this non-linear function of time. You have to sort of consider them together, and so we'll do that in just a minute.

Recipes for data preparation

All right, so we have this amazing patch code recipes, and what it does is it prepares your data for modeling. And you can start off by just giving it any data set, but we'll use the training set here. And we'll use a formula, and all it really does is, what the formula does is it says, things on the left side of the utility are outcomes and everything else represented by the dot is a predictor. So at this stage all the recipe does is says, okay, what's the role? Is it a predictor, is it an outcome? And it looks at the actual columns, and it measures like, are they category, are they factor, or strings, or ordered, or numeric, or integer. So it sort of just investigates the data and remembers what the columns are. Now the way recipes works is it's a little bit like the formula method or model.matrix

in R, but with a flavor of like dplyr. In dplyr, you would keep piping in things that you want to do to the data until you have it the way you want it. And the recipe works the same way. We're going to pipe in declarations of like, hey, I want you to log the data now, and I want you to do this with the data.

So the first thing we'll do is we'll create indicator variables for the day of the week by using a function called stepdummy. So those indicator variables are sometimes called dummy variables. So we're not mad at this variable. It's just like stepdummy is the name of it. And so all these things we're going to do and pipe in, they're called step functions. They all start with step under bar. And then the first argument in all of them is what variable should be affected by this. And you can use any dplyr selector here like you would with select. You could put in like everything minus something and starts with or whatever you want.

But at this point, what you're saying is I want you to take this column, make indicator variables out of it, then take it out and then leave the resulting indicators in there. There's a lot more you can do besides the dummy variables. Take a look at this chapter in the tidymodels book. It's something that we're very strong in for this package.

Oh, and one thing is, don't you hate what BaseR does? In this case, it would say like order day and then just take the level and just jam it up next to it. It's like the worst aesthetically thing ever. So what we do in StepDummy is you get to choose the delimiter and by default it's under bar. So the resulting Friday indicator would be order day under bar FRI. So you have some control about how it looks in your model results too. Alright, remember that though, that it's going to be named that.

What you could have done is recipes have a bunch, like a lot of extra selectors that you can just use here. And you can choose variables not only by their type, like are they character or factor or things like that, you can also choose them by their role. So if you want to do something to all the predictors, whatever might be in there, you can just say all predictors instead of listing them all out.

And so there's just a lot here that you can do with, and it's kind of a strong thing that we like about this because generally speaking, there are some recipe steps that will get rid of variables. So if you start writing specific variables down, especially late in the recipe, you may have eliminated them and it might cause an error in the recipe because you're saying, do this to variable x but variable x isn't in the data anymore. So we encourage people to use very generic selectors here.

Handling zero variance predictors and splines

Alright, so what's next? We have a lot of indicator variables, I think I call them item, and there are counts. But there might be one of those items in the training set where it just so happened they almost never get ordered, and the training set might have all zeros in it. That's not a big deal for some models, but for other models it doesn't like having a column of all zeros. So we call that a zero variance predictor.

So what this would do is when you use this recipe, it would look at your training set it would find any of the predictors I used all predictors, so it would search those and look for anything that has any column that has one unique value, and it marks that as you should get rid of that. And so this is a little helpful thing you can do. We also have something for what we call near zero variance predictors. Imagine you have like a thousand rows and you have 999 zeros in a single one, right? That's maybe not what you want in your data set. So we have some other ways of doing this that are a little more general.

So then what we're going to do is we're going to do splines. We're going to do exactly what GM smooth does. We're going to take our original column of the hour that the order came in, and we're going to generate as many columns as we want to make it non-linear. And if we add 100 columns it's going to be a really wiggly line. If we add very few columns, like 2 or 3, it's going to be a relatively smooth line. And so, you know, we don't know how many to add.

The typical parameter for that is called the degrees of freedom of the spline. And I'm just going to add 5 columns to represent that non-linearly. Why 5? I just picked it out of the air. It's not, I mean, looking at this, it's not like a super wiggly point here, or a relationship. 5, I'd probably do fine. But we have a variety of other basis expansions. There's a lot of different ways you can do splines. There's polynomial expansions. So there's quite a lot of ways you can represent a column in a non-linear way to the model you're going to use.

But you might not, you might be saying, like, I don't know about 5, though. I'm kind of stuck on that. You don't have to actually pick it. You'll see later that you can actually tune these options. So I can give it a non-value of this function called tune. And all that does is it just marks it to say, like, hey, later on, I don't, you're not going to be able to do this recipe until you know what this is. But we'll do it in a way that we'll search over different values of that parameter to figure out what the best performance is. So you can mark pretty much anything in here as being tunable and then just optimize it however you want to.

Interactions and normalization

So, here we are. We have, remember we have dummy variables that are, like, order day in the factor level. When we make spline terms, the way they're going to be named is they're going to be, like, order time under bar 1, under bar 2, under bar, all the way up to 5. So if you want to add interactions, and I looked to see if you can do this in BaseR, and you kind of can.

You can make interactions between a factor and a spline expansion, but you can only do it for one variable because it gives the splines the same names if you did it with a second variable. So we can do it for one variable. We can do it for pretty much everything, and the way we would do that is something called step interact, and then you can just give it whatever interactions you want to, and those interactions could use starts with or any dplyr selector. So we know all our indicators start with order day. We know our spline terms start off with order time, and that will let our spline terms, those relationships be different for day of the week, which it looks like they are. So this is something that would be a little bit hard to do with BaseR, but it's actually pretty simple to do it this way.

Alright, and then the last thing we'll do is, just for kicks, just to show that you can do it, is there are some models like GlimNet or k-nearest neighbors or something like that, where if you don't, they're going to want all the variables in the same units. If you think about computing a distance, if something is in square meters and another thing is in low counts like 0, 1, 2, 3, those aren't in the same units, and the variable with the larger numbers is going to dominate the computations. So what StepNormalize does is with your training set, it estimates the mean and variance of every column, and then when it goes to process the data set with the recipe, it subtracts the mean and divides by standard deviation, so everything has a standard normal distribution. And we're going to use GlimNet in a minute, so, spoiler alert, we need to do this for it.

Building and running the model with Parsnip and workflows

Alright, so this is the things we do to our data before we put it in the model, and then how we make the model in tidymodels. So what we do is the package that does that is called Parsnip. Parsnip has different model functions. So it's sort of almost like a two or three step process here, is you say what model you want to use. So the first thing you do is you say I want to use linear regression. And all that says is my outcome is a number, and my predictor should be like slopes and intercepts. That's pretty much all it says. It doesn't say how you're going to do the estimation, it doesn't say what those values should be or what not.

There's different modeling functions, so like nearest neighbor model is very equation-wise, like very different than linear regression. There's model functions for random forest, for decision trees, there's a whole host of them for time series models and so on. So step one with tidymodels is tell me what you want to do. Like what is it that you want the end result to be? And then the second thing you say is well how should I do it? And the how that we do it is called the engine. And so the default engine for linear regression is LM, but I'm going to show it here. You would just pipe into that linear regression function, set my engine to be LM.

And, you know, we could do that but, you know, maybe not. We could also use like TensorFlow. So maybe you want to use some sort of weird objective function through TensorFlow where you have a lot of data and you want to batch it. You can do that too. Or if you're more of a torch person, we have a torch modeling package called Brule. You could do that. If you have a lot of data, maybe you want to use Spark. You might have your data in a Spark cluster. So you can fit that model using Spark. If you're like me and you're really into Bayesian models, you can fit a linear regression using Stan. So that's not doing maximum likelihood, it's doing like Bayesian estimation using priors and all that stuff.

The thing we're going to use is GlimNet. And GlimNet's a really nice model. The package will be a little bit frustrating, but it's an excellent model that can handle a lot of different things. It automatically does feature selection for you and has a lot of niceties to it. So we've set our engine. We're ready to go.

It does have some tuning parameters. And these tuning parameters are things that you need to set up front. You can't directly estimate them like you estimate slopes and intercepts. You have to say up front what they should be. Sort of like the degrees of freedom. It's something that matters a lot, but there's no equation that tells you what the value should be. And the first one there is called the penalty. So GlimNet is a regularization method. It penalizes your coefficients. And so usually that penalty is between 0 and, I don't know, like 1. So I'm going to just pick .1 here just to show it to you.

And then GlimNet can also penalize in two different ways. It can do something called ridge regression. And it can also do something called lasso regression. Or it can do some mixture of the two of them. And so by saying mixture equal .5 here, I'm going to say give me a 50-50 mix of the ridge penalty and a 50-50 mix of the lasso penalty. And so these are just things that you would have to figure out which one worked better and which ones don't.

And also if you've ever used GlimNet or seen it, we're avoiding a lot of jargon here. Because in GlimNet they call these lambda and alpha. And so unless you're like reading up on the statistical literature in the original papers, like the help file tells you that. But if you look at the function definition and it's lambda, it's like, what is that? So we try to, as much as possible, avoid jargon. Whether that's a Greek letter or something that's just really specific to that package. We try to keep all the names the same of arguments and try to avoid sort of like things that maybe people wouldn't know what they are by looking at them.

Combining recipe and model into a workflow

Alright. So we know what we want to do and how we want to do it. So let's actually do it. Now we have a recipe that's going to process our data and give it in a format that we want to GlimNet. And so we can do that in like a couple steps. But the best and probably most straightforward thing you can do is combine them together in something called a workflow. And if you've ever seen like a Scikit-learn pipeline, it's basically the same thing. We're basically binding a preprocessor, which is the recipe, to a model and use them as one unit. So by adding the model and adding the workflow or adding the recipe into this workflow, I can do everything with this one object.

So if I want to fit my training set to fit my model to the training set, if you just use this with the workflow, it processes the recipe the way it's supposed to. It knows what the predictors are. It knows what the outcome is. It knows how to hand it off to the modeling function. And then the modeling function knows, oh yeah, GlimNet needs a matrix, so we're going to make sure we give it a matrix and does all the things that GlimNet needs. And then just as easily, the predict method is one simple step.

We're just predicting the first five rows of the validation set here. And you can see it comes back as a table. Most modeling functions will give you back a vector unless you're doing multivariate, in which case it's probably a matrix, or if you're doing classifications, sometimes it's a vector, but sometimes it's a matrix if you're getting probabilities. And the point I'm getting at here is we want everything in data frames, even if it's a data frame with a single row. Because you can very easily just bind calls this to your original data. You can do a lot more with a data frame if you're into the tidyverse, than you have getting output out. You don't know if it's going to be a vector or a matrix or a data frame. So we always do everything with data frames, and the naming of these things is very consistent. So you'll always know the names of what you get out.

Evaluating model performance

Now, what I can do now is if I want to see if this model works well in the validation set, I can predict the entire validation set, C-bind that to my original data where the outcome is, and then I can look at one of the tidymodels functions that computes the root mean squared error, and do all that. But we don't want you to have to worry about that. We have a function, a suite of functions that do that.

So using validation is like a single resample. So the code that you see here is the same code that you would use if you're using cross-validation or time series resampling. What you do is, basically, it's called fit resample. So we have a single one, and you just give it your resampling object, which we defined early on, and the workflow. And then it'll just fit, whether it's 10 models for 10-fold cross-validation, it's just one for this validation set. It'll do that for every tuning parameter value you have. We only have one right now. And then get the predictions and the right data, compute the performance metrics, and give those back to you.

So by running these three lines of code, you get an object back, and if you want to get the performance metrics out of it, you can see here the two that come back automatically. The root mean squared here is about 2.3 minutes, and the R squared is about 88%, which is not bad, right? You might be like, wow, that's pretty good, right?

But as I said, the only way to be comfortable with your data is to never look at it. So we've also developed a bunch of calibration and diagnostic functions for a lot of models in this package called probably. So if I load the probably package, there's a function called calplot regression. There's a bunch of calplot methods, and that just does the observe versus predicted, and keeps the axes the same, and aspect ratio of one die, and then it puts a little smoother on there, and you can see that actually the fits are pretty good, unless you're really going to take more than 40 minutes, and then we're really, really under predicting the data.

So that R squared of 88% might look pretty great, but maybe not once you look at this. So you might want to add some more features, and you might try a different model to figure out how you can do better on this. But the statistics don't always tell the entire story.

But the statistics don't always tell the entire story.

So what about parameters? I keep talking about tuning parameters. So at this point, I might say, well, maybe I need 10 degrees of freedom for the spline, or maybe I should use a lower penalty or mixture. At this point, what you can do is you can mark all those things for tuning, and then use our tuning machinery to optimize those values to get the lowest root mean squared error. And we're going to do that, but we're going to actually talk about a different model really quick. I'm kind of running late, so I'm going to go through this a little bit. There's this amazing model that not many people know about called Cubist, and it's been around for quite a long time, and it's a little bit like a boosted tree ensemble. What it initially does is it creates a regression tree. In this case, I've made a really short

regression tree on this data set. You can see there's four terminal nodes. And what a rule is, is it's a distinct path through this tree. So this is a model with four rules in it.

So what Cubist does is it gets these initial rules, and then with each one of these data sets at the end, it fits a linear regression to it. So now I have four linear regressions from my model. And then, like boosting, it builds a sequence of these models together, where at the end of this ensembling process, you might have 100 rules.

And you go to make a prediction. That prediction might only matter to like 20 of the rules. It gets their linear regression predictions and then it averages them. And it's a very, very effective model. And if you were wondering what these rules look like, in this particular case, here's a rule where the order time is less than about 14 minutes, and if it was on a Friday or Saturday and it was within, let's say, five miles, then you should predict it using this model. Or if I had a longer running order, or, I'm sorry, it was later in the afternoon on a Saturday, you know, I think you should use this model to predict.

So in Cubist, you get like maybe, you know, a subset of the total rules for every prediction. It makes a bunch of linear regression predictions and then it averages them together. And that tends to work pretty good. But we don't know how many things we should be averaging. We don't know if we should just do a little or do a lot. And so what we're going to do is tune that.

So that model is available in one of our packages called Rules, that has a bunch of rule-based models. It's called Cubist Rules. And then the variable that is the ensemble size is called Committees. So I'll mark that for tuning.

You know, Cubist is easy. You don't have to do any indicator variables. You don't have to do any transformations. You don't have to, I think it deals with missing values. So I'm just going to give it, instead of a recipe, I'm going to have my workflow take a formula and then go on my way. So I'm not going to tell it how many committees to use. I'm going to say I want 25 of them, and you just pick ones based on everything you know about this variable, this tuning parameter.

So instead of fitResamples, I'm going to use gridSearch, but the code is almost identical to what we did before. And it's going to use that validation set and training set. It's going to do this for 25 different values of committees, measure performance using the training set, and give you the metrics back.

And when it does that, it looks like, looking at it, it's like, oh, it wants a lot of rules here, and the RMSE is, you know, I'd probably say substantially lower, not massively lower, but it's an improvement on the linear regression model. I can auto-plot that and see what that trend is. Well, it turns out that we don't actually need this many. We could probably do with like 10 or 12 and get a simpler model and probably get about the same level of performance.

So then, you know, so now I think I know what kind of Cubist model I like. You know, if you plot this, the data points, you can see, oh, yeah, I like this better, what not. Spoiler alert, this is a better model. And then we have other machinery that I'm not going to go into too much where you can finalize that model, say, oh, give me a Cubist model with, you know, 10 committees, and then we have a nice function called last fit that it'll take that initial fit, it'll run the model on your entire training set, fit the testing set, and give you the plot, the data for plotting the metrics in one easy step.

Wrapping up and what's next for tidymodels

So that's like the gist of what tidymodels does. I'm just trying to give you a flavor of like what the interface looks like and the ways we think about this. If you're just doing something super simple like, you know, empty cars, you know, you might just stick to LM and nobody here is going to be offended by that. It's not for like I need something really complicated, but there are oftentimes there's things you want to do that are just kind of a pain in doing them in base R or in the packages, so it's really designed to help with that a lot.

There's just some links here to things in the tidymodels book. I only have like two minutes. The things that we've done recently is hopefully before the end of the year we'll release all of the packages that have the integrated survival analysis, so you can do everything we just talked about using like proportional hazards models or tree-based models for sensor data and so on. We spent a lot of time, it turns out case weights are really complicated and nobody really knew that so we enabled that in a lot of places for tidymodels and I just spent a lot of time decoding what these things are called conformal inference tools, so those are in probably.

The things we're working on right now is model fairness metrics and ways to build your models to sort of mitigate any fairness issues. If anybody here does causal inference, we have some sort of prototype causal inference tools so if you're interested, let us know. We'd love to get you to look at those. And then one thing we want to do is post-processing. If you want to calibrate your model or optimize the probability threshold to say whether something's an event or not we have a list of things that we want to allow you to do or even optimize that come after the model fits. We have pre-processing the model, now we have a limited number of post-processing tools that we want to do.

And of course the most important thing, thank you for the invitation to speak. Everybody today was sharing their mate with me. It was very nice. Word got around, I was like, take mine. And especially thanks to the group. They do amazing work including all the people who used to be in the group or have been interns or helped out. It's been a huge team project and I really appreciate it. Thanks.

Q&A: reproducibility and randomness

Questions?

Thank you for your presentation. tidymodels is really big and I would like to know how, because in the process to do and compare models you are doing a lot of steps and you have different randomness in some points and different the models also have some randomness inside and how you are thinking in the reproducibility when you are comparing different models that have different levels of reproducibility there. I know that you have something saying about here we have a problem about reproducibility but I don't remember where.

So the effect of randomness on the models, the first thing I would say is that will come out in resampling. So if you fit the model once and you don't know and the model uses random numbers was that a good set of random numbers? You can get the effect of random numbers by doing cross-validation so that will measure the effect of it.

The usage of it is we have inherited a lot of things that I really haven't dealt with in Carrot. I came in with a whole list of okay we know what not to do a few things. For a while we had to figure out how when you are running in parallel to make sure you set the seeds inside the parallel workers to make sure those get set right. I have to say that with one or two exceptions everything is locked up pretty tight. If you set the seed before you do model tuning or something in 99% of the cases there is no issues with well the problem is like I said R often farms a lot out to other computing languages and the one I will tell you about one that is well known is if you are doing a support vector machine the main package that we use for that uses a C library called LibSVM and it turns out that I am not sure why, Crayon usually frowns on this, I am surprised this is a thing but it has its own random number generator in there. So if you are fitting a classification model, especially if it is one that has really low event priorities or event prevalences, if you run that model twice you are always going to get different answers but you get noticeably different answers in some cases. So that is something external to us and I have over the years emailed the maintainer and they are like we did not write that library.

But the other thing that we realized is with deep learning we are trying to do a lot leverage a lot of what comes with TensorFlow and Torch, they do a lot on the GPU and I am getting the idea that it is really hard to set seeds in a GPU because none of them do it. So we try to lock that down as much as possible in the R packages, talking to the people who manage those there were ways to set it by setting system environmental variables but they have over the years given us ways to set that so we have done as best as we can but chances are if you are using a GPU you cannot really guarantee that that is going to happen. But I think for the most part it is pretty solid in terms of reproducibility.

But I think for the most part it is pretty solid in terms of reproducibility.

Q&A: Spark support

You mentioned that one of the possibilities when setting the engine was using Spark as an engine. I would like to know how transparent is the transition for the user when using a vanilla engine, let's say Random Forest, the more traditional implementation, if I want to switch to Spark how much modification of my code do I have to do or is more transparent like just switching the engine?

I've got good news and bad news. Which one would you like first? Bad news. Right now in setting models you can only just fit like with Parsnip a model with Spark. You can't tune them or resample them. And the reason for that is Spark DataFrames are really good at what they do but they are kind of hard to program against. So for example Spark DataFrame doesn't really have a way to tell you how many rows are in it because you would have to evaluate it to know and in some cases we need to know what that is.

And so I think if you're using Spark, is there anybody else here that uses Spark a lot? Okay. So if it's of interest to you, what I would suggest you do is you go into Parsnip and put in issues about the tidymodels package or something like one of our repos and say we need more comprehensive Spark support. So the bad news is you can't do a lot with Spark and tidymodels. The good news is it's really easy to know when that transition happens because there's no transition.

So there's somebody who works with us a lot named Edgar Ruiz and he's like the Spark genius. He probably knows more about Spark than most people on the planet and I think he has grand designs for how you could make Spark work in almost everywhere in tidymodels. The problem is that complexity, we don't know how many people — we don't know what the denominator is, or we don't know what the numerator is. We know how many people roughly use Spark but we don't know how many people out of that want this and it would be quite a lot of work especially for him and we don't know if it's something we can do. So if there are people out there that would like something like that you need to be vocal about it to us.

Can I give you feedback about that topic? In my organization I think that chicken and egg problem that idea would be more developed. Like being able to code a model with a single implementation and then porting it to Spark it would be like a killer application for many organizations because even for people to use Python it might be rare. Coding in Stack Overflow is very easy but the transitioning, the recoding of the Stack Overflow model to Spark it's more complicated, so that kind of portability would be very important. I didn't know it was possible until you mentioned it now.

There's some limitations though, so for example let's say you're doing — you need to impute data, like you have missing data and you need to build an imputation model into your preprocessor to predict the missing values, like if you're doing nearest neighbor imputation now you've got to save the entire set at that point in the recipe, so there's things that I'm not sure they're even feasible to do even if we try in Spark. So if we end up having more substantial Spark support we don't know how to do this quite yet in documentation, we're going to need to tell you when you can use this and when you can't. We can have the function tell you when you execute it but we need to figure out where that boundary is and how to document that boundary so that you're not frustrated like you see there's a step to do something and it turns out it doesn't do it and it gives you an error. We're not really sure how to organize that.

We have this problem with data.table, we have the same thing with arrow, like in some cases outside of the Spark modeling functions all that has to be converted to a data set at some point or a data frame at some point to use in standard R libraries, so this issue of the data storage versus the training set in the model is a really tough one, but I know we can do a lot more with Spark.

Q&A: recipe diagnostics and feedback

Thanks Max, one thing that I really like about the recipes part is generalized additive models, GAMs. I found that really hard to teach GAMs but now I find it easy because I say this is actually what it's doing, you put the smoothers in the or the splines in the recipes and then you can actually inspect them when you get your model results as well. But the thing that I struggle with with recipes is a lack of feedback on what's happening at every step and a lot of these things are really vital and can be critically damaging to your data prior. It's a little bit black boxy. I'd like to have more diagnostics come out from each step. I used a recipe in a workflow and I said fit and predict and all that stuff just happened which is opaque.

If you're using a recipe directly, the function that estimates the recipe has like a logging — it does have a logging process that you can trigger and it'll tell you, especially it'll tell you like this many variables went in, this many came out, I eliminated these. We should definitely try to add that option during resampling and fitting, so we would just need a way — I mean we have a mechanism, we'd have to add that potential for that flag to go through to the actual recipe execution. You're not the first person to say that, so yeah that's something that we could definitely do.

I'm pretty sure we can do that I mean it can be done but it just depends on how much Emil is going to regret it yeah

so we have time for one last question

Hi in the documentation in the parsnip fine models later or a few months a year ago there is one option for a model in linear regression in one page we saw all the engines now each engine I think users that don't know what engine do that need one extra page so when I want to use a new model I go to find parsnip I see several pages so I don't know what to do I think there was useful one page that this engine is for that when you want to do that

congratulations very nice package we are your fans thank you

do you want more information on a description of how the model works, like the math of the model or how to use the code what more would you like to know in that instance for example sometimes there is in linear regression there are various engines what engine I have to choose in what thing I decide what engine to use if I don't know what to do in each package so a page in that more guidelines

I personally hope in the next coming year I will be writing a lot more public web documentation on how these models work before that happens he comes out to the tidymodels and I'll just pick this one it's the first one you can search here for the type of model we used actually I'll look for cubist

so cubist, it tells you what package it is, what the engine name is there is a little link here it takes you out to what we call the engine documentation

so did you get here? yeah one step before we saw several slides we need to do several pages and more information

so you need something before this to source DRU ok ok Hannah's got an idea can you show the parsnip reference page that's this? no do you mean like the linear reg one? yes, just for the model definition right because it lists the engines let me open R and do that you can just go to parsnip.tidymodels.org

this is what I live with this is what he pays me for that's right, I pay for this oh my god let's talk about linear regression I'm going to blow this up if I do linear reg so if you're not at the website if you're in R here's the main page that lists all the myriad I didn't know it was that many for linear reg one nice thing is it tells you what packages this requires an extension package this is the page you were looking for? Hannah? yeah yeah

it still exists on the parsnip page it does so if you're in parsnip and you click on reference it lists you to model definitions and then if you click on linear regression you get to that on the package downside as well

we could fake that table out and say make an artificial all engines and have it just link to this we will try to put many ways to the right thing we should just do it now let's make a pull request no we're not that's really good feedback

if I a newbie in modeling I saw linear reg and there is many giants as a real user we know which engine choose a newbie know I think it's confusing more than too much length

one other thing I'll show you that might help in the future is I'm going to go to the tidymodels book to the website for the tidymodels book and in there we have a nice table it's like an appendix that if you're going to fit one of these models you might be wondering do I need to center and scale do I need to impute and so we have a table here that gives you the list and says yeah you should definitely need to create indicator variables for this and definitely need to maybe normalize some things it's not a model thing but it's like even before you get to the model you'll kind of need to know this and it's not obvious sometimes we put some of that in the engine documentation thank you very much Max

Closing remarks and community acknowledgements

well ok exciting moments are coming from the end thank you very much for putting up all this time it was a pleasure, right? a pleasure it was for me, I always have the feeling that I want a little more now I'm a little tired it was good to have you here to have all the guests thank you very much for coming for participating for continuing to build the community

a lot of nice things happened a lot of people came Diane came, Max came, Hannah came people from Appsilon came who gave courses Riva came, who also gave a course we learned and we enjoyed the presence of people who come from far away, who have a lot of experience that is also very good we took our photo of our ladies there we are several times because there were people who did not come

and well, we had our moment of stickers which is a very stressful moment for people I saw them fighting and jumping like crazy well and also some difficult moments but it's really good to see the friends and spend time together

obviously this requires a lot of effort and a lot of work and basically with little budget it requires a lot more work so I love to see them, to work with them it's been years since we've been working on this and trying to build the R community in Latin America and I think we're doing pretty well

it's been years since we've been working on this and trying to build the R community in Latin America and I think we're doing pretty well

I put on a suit we put on a suit 12 the maximum I think it's a 7 a 7 well, we're going to lower the note a little well

if you wonder where I was in the photo that they are crying I was on the beach with my children, enjoying I was the most well as we always say we're talking about community and that's done with all the people who contribute their work their time, their ideas, their energy to do this

so thank you very much again to this year's organizing committee Nacho, Ignacio Álvarez Castro Mauro Loprete, Andrea Gómez Varga Beatriz Mirs, Juan Pablo Ruiz Nicolini El Tucu, Macarena Quiroga Jessica Formoso, María Nanton Roberto Villegas Díaz, Luis Pao Elio, Patricia Fabricio, Estefany y Bruna

also thank you very much to the scientific committee that helps us to review all the proposals you send and decide how we are going to present, in what format thank you very much to Sara, Marcela, Ignacio Virginia, Zulema, Naim Elio, Jocelyn, Marina Cristina, Paola, Juan Cruz Andrés, Felipe, Alejandro, Denis María, Adriana, Carolina Juan José, Jennifer, Juan, Leonardo Ricardo, Estefany, Francisca Felipe, Vilma, Germán, Gabriela, Alejandra Inés y Luis Un aplauso para todos ellos

Para todos los que están acá y se están preguntando qué cool sería participar, sí son todos bienvenidos muchas gracias a nuestros sponsors a la facultad, al Instituto de Estadística de la Universidad de la República de Uruguay al Art Consortium, a Absilon a Chicada y a nuestras comunidades amigas ArtLadies y ArtOpenSci y muchísimas gracias al equipo local de la facultad Laura, equipo de registro equipo de informática, electrotecnia comunicaciones, vigilancia y adecanato gracias totales

Looking ahead to future conferences

Hablar del futuro a mediano plazo, a largo plazo y a corto plazo A mediano plazo qué sigue después de una conferencia uno sale con mucho entusiasmo con ganas de hacer cosas y conocimos a muchas personas en estos días y la idea es mantenerse en contacto quienes no lo han hecho aún, únanse al Slack de LatinR entre las conferencias se mantiene muy activo la gente comparte proyectos comparte dudas recursos, o sea, un espacio que se mantiene activo, no solo existe durante la época de la conferencia

La comunidad R en general ha sido muy activa en redes sociales, ese también es un espacio para mantenerse en contacto con las personas que conocieron hoy día y también están los Meetup locales de ArtLadies, de los grupos de usuarios de R muchos de ellos estuvieron medios inactivos durante la pandemia, espero que para muchos este sea el entusiasmo para partir

¿Dónde está Claudia? Ya estamos como viendo que la gente empieza a organizar eventos ¿Qué va a pasar en 2024? En 2024 vamos a hacer la conferencia en línea hay varias razones para esto la última vez que la hicimos presencial fue en Chile, el año 2019 y ese año una persona de academia por inscribirse a la conferencia y a dos tutoriales pagó alrededor de 220 dólares este año una persona que se inscribía de academia pagó 20 dólares por la conferencia y pagó 15 dólares por cada tutorial, o sea el precio es mucho menor

y eso es posible porque estuvimos varios años haciendo conferencias en línea que nos permitían juntar presupuesto para hacer la conferencia accesible este año por primera vez pudimos hacer que todas las presentaciones el presentador o presentadora principal no pagara el ticket de la conferencia es importante, obviamente nos encantaría organizar la conferencia presencial todos los años pero este costo tendríamos que traspasarlo a los tickets para que la gente se registre y eso lo que hace es convertir la conferencia en algo inaccesible sabemos que existen muchas conferencias que tienen precios que son el sueldo mínimo de nuestros países y eso es inviable

entonces queremos también mantener hacer que la conferencia sea accesible y la idea va a ser ir intercalando versiones en línea con versiones presenciales para poder mantener los costos de una manera que sea viable para que las personas puedan participar ¿qué significa? que el 2024 nos vamos a volver a reunir pero no nos vamos a volver a reunir de manera virtual y eso también le da la oportunidad a personas que no pueden viajar porque cumplen labores de cuidado porque tienen trabajos que no les permiten viajar o por presupuesto puedan también participar de esta comunidad entonces esa es una decisión importante

a largo plazo respecto de lo que va a pasar el año 2025 que va a volver a ser presencial queremos escuchar propuestas de a quienes les interesaría ser la sede de la edición de la edición 2025 sabemos que ya hay varias personas en distintos países que tienen interés vamos a dejar un formulario en el Slack de LatinR y en la página web de LatinR para conocer su interés respecto de ofrecer su ciudad para que llevemos la conferencia a ese lugar y en dos años más nos volvamos a encontrar

de manera presencial y a corto plazo el sorteo ya tenemos varias cosas para sortear todo el mundo estaba preguntando por los stickers del Tinyverse y solo habían dos esos dos así que los vamos a sortear para que se ajuste a la distribución tenemos algunas dependiendo de su país remera, polera, sudadera que vamos a sortear también

y lo que todo el mundo estaba esperando la segunda edición del libro Art Packages autografiada por ambos autores Jenny Bryan que fue keynote speaker en LatinR el 2018 y David Wickham que fue keynote speaker en LatinR el 2019 así que vamos a hacer el sorteo con R tiene que estar acá así que me voy a mover hacia el otro lado

El sorteo

redoble de tambores

ya vamos se van los stickers Natalia la función sortear nos borramos Joaquín Ferreira ¿está acá? al agua ¿no dicen al agua? en Chile decimos al agua María Fernanda González al agua

el otro sticker Roberto Aguirre ¿al agua? Victoria Landaberry bueno, tenía efectos de sonido pero no sonó así que no importa Valentina Cortés Carolina Musso se fue Lucía ¿está Fernaberry? Mauro Lepore

Emilio Aguirre Daniel Araujo Bruno ¿está por ahí? Bruno Tancred Oscar Montañez Mayara Andrade Negli Gallardo ¿está aquí? Giovanni Reales Giovanni está llegó justo para el sorteo eso es timing

Noelia Castellano se fue Bernadette y ahora Redomble de Tambores ¿quién será? ¿quién es? Santiago Nicoleta ¿no está? Federico Rodríguez Valentín Romanelli ¿sí? ¿es él? ¿sí?

bueno, muchas gracias y nos vemos virtualmente el 2024 y en algún lugar de Latinoamérica el 2025

Subtítulos realizados por la comunidad de Amara.org