Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Lab. I'm Libby. I run community here at Posit, and I'm joined by Isabella Velazquez. Isabella, would you like to say hello?

Hi, everyone. Thanks for joining.

I am so excited to be joined today by our lab manager, our guest lab manager, Davis Vaughan. Davis, would you like to say a quick hello and tell us who you are and what you do?

Sure. Hi, everyone. It's nice to be here. I'm a software engineer at Posit. I work on the Positron teams and the tidyverse teams. I work on dplyr, tidyr. I also am the co-creator of air, the R formatter.

Nice. All right, Davis, what are you going to be showing us today on the lab?

So, yeah, I figured today we could talk about dplyr 1.2 a little bit. It'd be kind of fun to just kind of go over some of the new functions, like filter out and recode values and replace values, and then maybe also a little bit about like how functions in the tidyverse are lifecycled. Like, what does it mean to be superseded? What does it mean to be deprecated? And what is the difference?

How do decisions get made? What kind of conversations are had behind the scenes? That's what I really want to know. I'm going to ask some nosy questions about that.

Another thing that we will briefly touch on is also like how – well, not briefly touch on. This is going to be the theme. How community-driven development happens and why community is such a huge part of the development of open source packages.

And also, Davis will talk a little bit about wife-driven development.

Introducing recode values

All right, everybody, let's get ready to watch Davis share his screen and walk through a couple of things for us, the first one being we have a new function called recode values. This one's really exciting for me because I have had a janky workaround to recode values for a long time, and I'm not the only one who had that janky workaround. Crystal Lewis also had that janky workaround, and we've all been sort of circumnavigating a slight hole that was in the dplyr suite of functions.

So, Davis, would you be willing to walk us through a demo of why we needed this new function and what it does?

Do you happen to have the BlueSky post from either you or Crystal just about the issue?

You know what? I can go ahead and find it, yeah.

Yeah, well, we'll find it. There's a BlueSky post where somebody had posted – if you're not on BlueSky, get there. The RStats data BlueSky scene is really, really great. Somebody posted a question like, hey, how can I recode these values based on a named list, I think is what it was, right? And everybody was chiming in and being like, oh, we can do different things. I'm like, oh, here's my workaround, and Crystal Lewis was like, me too, I have the same workaround.

Yeah, so I'll just show you.

Yep, that's it. There's my screen chat.

Yeah, so we were all, like, super confused about this whole idea. It's not going to look very pretty for a second.

That's okay.

So, like, I have a named list here, right? This is like a Likert encoding, where, like, one means strongly disagree, five means strongly agree, and then there's everything in between. And you might have the data in the form of just the numbers, and you want to recode that using this mapping into the actual, like, word descriptions. And there was this whole discussion that went on of, like, what's the best way to do that? And the kind of hacky solution was using a superseded dplyr function called recode, plus this, like, triple bang type thing to get that to work. From Arlang. It's an Arlang operator.

Yeah.

But it's a very cool Arlang operator, okay? Like, it's still so valid. It's really, really cool.

It is cool. It is valid.

Wouldn't it be great to, like, have a way to just do this without that?

So, it got a lot of traction, and, like, it got us thinking of, like, why is this so stupidly difficult? So, we clearly figured out there was some kind of, like, hole here. So, the hole was filled by recode values.

So, just, like, a little bit of how this works, right? So, you've got your Likert scores here. It's just a column in your Tybl. And, like, there's some typical other way you might do this if you didn't have, like, a lookup table on the side, where you might go through case win, and you might say, you know, if score is one, strongly disagree. If score is two, disagree, and so on. And that works. I mean, it's wordy, but it works.

And, like, one of the nice things about this new function called recode values is that it takes this, like, common bit of score and says, hey, you can actually, like, pipe that thing in. We recognize that all of the individual cases are going to be comparing to score. So, you can pull that to the front. So, now it's score, and I want to recode all of the individual values from score, where one becomes disagree, two becomes strongly disagree, two becomes disagree, and so on. So, you can kind of get rid of this score equal, equal bit. So, that's nice as, like, one step in the right direction.

But remember, we had that, like, lookup table thing that we cared about. So, what we can do is actually, like, if you squint at recode values here, it kind of looks like a lookup table. Like, it's on the left-hand side, it's mapping to the right-hand side, and that looks pretty nice. So, what we can do is make that more explicit. Like, we can pull that out into a table called lookup. And instead of this, like, very familiar case-winning interface, we can use an alternate interface where you supply from and to columns directly, where I can say, from this lookup table, like, where it's one through five, map that to these words, strongly disagree, disagree, and so on. So, recode score from that to that. And that works great.

And that is kind of the answer to that, like, how do we do this recode bang, bang, bang thing? Like, why is this so hard? Like, we can now pull this really nice lookup table out separately and supply it in our pipeline, which is especially really nice if you have a really long pipeline, and things get, like, it just lets you focus on actually what you're doing rather than the mapping itself.

And the last thing I'll say about it before we stop is this little bit here. Like, it's very common, in my experience, for this lookup table to not be inline. It's actually from CSV file somewhere else. So, that makes, like, the recode bang, bang, bang even, like, it's, like, your only option because you couldn't actually inline it with the case win before. So, now, if you did have it in a CSV, it's just as easy as reading it in, supplying it here.

Community-driven development

Yes, and I want to reemphasize for everybody that everybody has a different type of need, right? Like, whatever your experience is of needing a function, most people need it in a different way. Everything is so, so unique, right? So, there are a lot of responses that are, like, why would I ever need this? I've never needed it before. You just specifically have not happened to need it, but then other people have really, really needed it, right?

So, if you've ever only written pretty short case win statements and you've only ever needed maybe three or four things in your case win, maybe up to, like, 10, 11, then it's possible you haven't experienced needing 800 levels of case win, right? Like, it's possible that you haven't worked with really wild survey data or really, really wild, you know, longitudinal data that has needed tons and tons of recoding. And in those instances, a thing like this that abstracts away all of the complications can be really magical because Davis is right. Those lookup tables exist on their own. In fact, I just sent Isabella some survey data yesterday that had a CSV lookup table all on its own that I created that had, you know, 40 or something entries in it. That's usually just the way that it goes.

So, the real thing that I want to talk about behind this with Davis is actually the different little bits and pieces that came together to tell him it was time to add a function to dplyr because it wasn't just that blue sky post and it wasn't just me talking about it, Crystal Lewis, making a blog post about it. Davis, what all was going on behind the scenes that came together for you to say, like, well, it's definitely time to take action? Because you'd already been thinking about it when I messaged you about it, right?

Yeah, we've had, like, we've had issues on the dplyr tracker for years of, like, you know, you superseded recode, but, like, you didn't give us anything better. Like, it just feels like there's some missing holes where, like, you superseded this function, but there wasn't a complete replacement. And, like, at the time, we just kind of, like, looked at that and we were like, did we? I don't really know. But, like, you know, then you see all this community impact of, like, yeah, you really did. Like, here's the exact example of, like, the hole that you're missing some kind of answer for. And that really, like, nails down the point of, like, yeah, we think that there really is a gap here. We think we could do something better. Like, we agree that recode probably wasn't the best API to begin with. We don't feel bad about that decision. But, like, we didn't fill the hole with something better.

Yeah, we've had, like, we've had issues on the dplyr tracker for years of, like, you know, you superseded recode, but, like, you didn't give us anything better. Like, it just feels like there's some missing holes where, like, you superseded this function, but there wasn't a complete replacement.

Right. So there was already stuff going on behind the scenes. And this brings me to GitHub repos.

Often when there's a feature that you want added to something, you will hear us in the sort of general open source community say go thumbs up that issue. It's the best way for us to know that you care about it. It's even more important than adding a comment. Right. If you add a comment on an issue that says this is really important to me, it's actually less impactful than putting a thumbs up. So, Davis, can you walk through what that looks like? Like, how does someone go thumbs up an issue? And then how do you use that to filter and find things that are valuable to the community?

Sure. I have a good example, actually. So I work on Positron. Positron is the new data science IDE from Posit. And, like, we will go over in the issues list over here in Positron. And, like, something that we do, like, fairly often is go over here and sort by the number of thumbs up. And you probably won't be surprised if you're using Positron often that Quarto inline output is the number one thumbs up issue with over 106 thumbs up. So if you wanted to add your support to request this feature, you could hit the little smiley face here and thumbs up. And then it would bump to 107. So, you know, I really want Quarto inline output. So let's just do that. And we use this a lot. Like, we go in here and we sort by, like, you know, how many thumbs up. And this is kind of some kind of impact or some kind of guide of, like, what does the community care about.

Perfect. All right. So, everybody, the main issue, post, reaction, add a thumbs up. That lets people know that you're really excited about it. And I really encourage you to go check out Positron's GitHub. Go look at all the open issues. But also go check out Quarto. Go check out all kinds of things.

Audience questions

I want to hop over to a couple of questions. Mike had asked if you could show us how packages like dplyr are created along with functions, it would be helpful. Because there are a number of moving parts to the package. We're not going to go that deep today, Mike. We're going to talk about more surface level stuff, like adding new functions to an established package and deprecating functions. But that's a fantastic idea. We did have Kylie Ainsley on a few weeks ago who talked about a little bit about our package. Not package development, but the package structure, at least. I think it would be a great idea to talk more about packages in a future episode.

We also had a question from Nathan that was, so, if we have a data dictionary, we could use recode with across to apply the right values to the right question? You can put, you can use almost all of these with across to go across all of your questions. So, you could do across all or you could do something like starts with, right? Like to grab all of your columns that start with something. Let's say all of your questions in your survey start with like a question underscore or Q underscore. Then you can, yeah, grab all of those and apply it all across those and mutate. And we might be able to show that with Davis as well.

I was trying to find that example that you asked me to add.

Yeah, I pushed Davis to add this because often when there's not something in a vignette that extends the function into using across or into using starts with or something like that, I find that learners, when I'm teaching, get really stuck. They're like, okay, I know that this function does this. I know that there's the ability to apply it across other columns, but I don't know how to put those pieces together. So, I think this is kind of what they're asking for, right? You have multiple columns where they're all Likert codes and they're all grouped by this common name of score. So, you could say if it starts with score, I want to recode those values using that table that we had before from two. And then it goes ahead and recodes all of those columns that match score.

Yes, perfect.

And then Noor had asked, why dplyr 1.2 as opposed to 2.0? Is there a limit, a threshold where you're like, aha, we've reached 2.0?

Yeah, the 1.0 release of dplyr was a chance for us to... Typically, these major releases like 1.0 and 2.0 are reserved for I really, really want to break something that really might affect a lot of people's code. We have made some seriously bad decision and we think that there's something much better now. And we regret something deeply and we'd like to break it, but we'd like to let people know that there's some big reason we're going to break it. So, it seems unlikely that we will ever have a dplyr 2.0 at this point. We feel mostly pretty good about all the decisions that we've made. I could regret those words so fast. It might happen. But most of the dplyr releases will be like 1.2, 1.3.

And anything that are really small, just bug fixes, will be like 1.2.1. It's not too much to worry about in a very small patch release.

All right, perfect. And did we touch on what a tidy up is? Not yet. You might have heard us, if we were talking on Blue Sky or we're posting on LinkedIn, you might have seen the phrase tidy up. Can you talk a little bit about what that means?

What are tidyups?

Sure. So, one of the big ways that we get feedback from the community, in addition to Blue Sky and LinkedIn and kind of talking with you all there, is the somewhat new-ish process that we use called tidyups. If you're familiar with PEPs from the Python ecosystem, which is like a Python enhancement proposal, tidyups are kind of our community-facing similar-ish idea. So, they live here under Tidyverse, Tidy Ups on GitHub. We have eight of them. And each of these is some kind of proposed large change to the Tidyverse. Either it's some kind of breaking change or some kind of totally new feature that's pretty big and we'd like to get it right the first time around.

So, for like recoding and replacing, for example, since we've gotten this wrong... Can you zoom in just a little bit, David? Oh, yes. Yes, of course. Just a little click, click. Yeah, awesome. Since we got this wrong so many times before, we decided to do a tidy up to get a lot of community feedback to make sure that we got it right this time around. So, you can go through and read this. It's just like a markdown document.

I'll show you the outline. Oh, actually, it's in here now. The outline is kind of just the most interesting thing to look at real quick. You can get a general vibe of our motivation for this function. All these examples of what you might have done instead. Here's how it's better than X, Y, and Z. And how are we going to preserve backwards compatibility? How do you teach this if you're a teacher? We try to think of as many things as possible in this tidy up of what could be interesting for people. And then we release it for the community to kind of give feedback on this markdown document. And a lot of people did, which is great. We had a lot of people come through and say all kinds of things they do or don't like about it. And people also gave a lot of feedback for this one on Blue Sky and LinkedIn, which we used as well. We kind of incorporated everything in here. So, it was great.

If you ever see tidyups out in the wild, this is your chance to weigh in on this package that we all use and that many, many people contribute to and maintain.

If you ever see tidyups out in the wild, this is your chance to weigh in on this package that we all use and that many, many people contribute to and maintain.

I really want to encourage everybody, if you've never, ever participated in open source development before, to go through and read things. Read the comments of others. If you are ever nervous about commenting, hop on the Discord and let us know. Like, hey, I really want to add this comment, but I don't know if this is the right thing to do. Ask somebody, and they can help you. I'll chime in and help as well. Or Isabella will, or Daniel. Somebody will hold your hand a little bit.

I wanted to point out something that Jenny Bryan said in the chat, which she was kind of adding a little bit more context to our conversation around thumbs-upping issues, which I think is really helpful. She said every time someone comments on an issue, every dev gets a notification. And it can be really overwhelming if you've got thousands of issues open for every single dev to be getting a notification for every single comment that comes through.

So it's useful if the comment really adds something, a new example, a new angle. But it's not as helpful if the comment is just, hey, I'm having this problem too. If that is the point that you're trying to get across, those thumbs-up are a better and more efficient way to do that. So thank you, Jenny, for the extra context there.

Introducing filter_out

All right. I say we go ahead and move on, Davis, to talking about the next function that we are adding to dplyr, that has been added to dplyr. Would you like to share that little demo of filter out? This one I saw had some controversy on LinkedIn and Blue Sky. People going, I don't know why you would need this. And then a bunch of people saying, oh my gosh, I've been waiting for this forever.

Yeah. In fact, let's see. We have issues from 2019 or something like that that was asking for filter out. And we were just like, no, we don't need this right now. We don't think we need this. We don't think we need this. We were the same as these people. We were the people on LinkedIn going, you don't need that. That's no good.

But eventually, there's an example that I'll show you here that kind of convinced me that we need filter out. So what is this idea? So filter lets you specify which rows you want to keep. Filter out lets you specify which rows you want to drop.

Yes. Can we zoom in really quickly one more time so we can get bigger? We're getting feedback on YouTube that's like, it's not big enough, y'all. We need bigger text. I totally get it. See how big we can get without annoying me. There we go. How's that? Pretty good. Yeah. Pretty good. Everybody can see, right? Let me know in the chat if you can't. Yeah, just let me know.

Let's say we have some patient's data here. This is just tracking whether the person is dead or alive. And if they're dead, when did they die, I think, or the date of this kind of information here. So like we might have some question where you want to know, you want to drop the rows where two things are happening at once. The patient is deceased. And the year of that kind of information was before 2012. So like two things. Patient is deceased. That's the deceased column. And the year was before 2012. That goes with the date column.

So you could like directly translate this into a filter statement where it's like they're deceased. The date is before 2012. And rather than keeping those rows, I'm looking to drop those rows. So you could like wrap this whole thing in parentheses and just put a not in front of it. That's kind of your most direct translation. And that kind of works, except it's not exactly what you want. And it's a little bit subtle to understand why this isn't exactly what you want.

But if you think about the question, like I want to drop like only when I know that that patient is deceased. And I want to drop rows only when I know that the year was before 2012. So if you look at the results here, what you'll notice is that my wife, Sarah, has been dropped out of this kind of data set here. Because we don't actually know that she's deceased or what the date was. But it has dropped her out of this data set kind of by accident. We dropped too many rows. And the reason for that is kind of complicated, but it has to do with how filter works with missing values.

And what you end up having to do instead, you just kind of start adding in these like really, really ugly like deceased and it's not in A. It's before 2012 and it's not in A. And like you end up kind of squinting at this for a while and maybe you end up with the right answer. You're like, okay, now I finally got like what I was looking for. Like if I don't know the answer, I still want to keep that row. And typically like that's what you're actually after when your question, your kind of problem statement up top says like I want to filter out specific rows. Like you typically don't want to drop the missing values.

So what we've ended up with with filter out is a way for you to kind of directly translate your question right into code as it was before. And keep that intent of I don't want to drop the missing values either. Like if it's this lets you say I only want to drop the deceased and I only want to drop where I know date is before 2012. Since you don't know that about say max here, that row sticks around. Hopefully that makes a little bit of sense. But the general guideline at the bottom here is if you decide you only want to keep rows, like if that's how you're framing the problem in your head, you want to filter. If you're kind of thinking about it as I want to get rid of some stuff, I want to drop rows, you probably want filter out.

Perfect. I'm going to check the discord really quickly to see if we have any questions. We did have one from Luis earlier that was saying like, hey, I didn't catch that anonymous formula notation in the last shared screen. What was that for? But I'm not sure, Luis, what you were looking at or what Davis might have been showing. If you have more context on that, let me know in the discord and we can possibly go back and look. If it was on GitHub, we can go back to GitHub and look at stuff.

But I wanted to, Davis, if you could share your screen one more time. We had gotten some questions and feedback about the way that the example is styled, right, where it's like filter out deceased. It's not deceased equals true. It's just deceased. And then also, what's the difference between a comma and an and? And what's the difference between having a comma versus having two filter out statements where I'm like filter out deceased equals true. Filter out date greater than 2012. Could you talk about that just a little bit?

Yeah. So in this case, these are equivalent. Anytime you have a comma in either filter out or filter, the comma is kind of a stand in for and. But whenever you have like multiple conditions, like they kind of look nicer, like especially as you add more and more conditions. If they're separated by a comma, because that means they kind of end up on their own line and they're kind of separated. And like you could think about them individually rather than having a whole bunch of and signs here.

If you wanted to do an or instead, then you could either use that deceased or the date is less than 2012. Or you could use the other one of the other new things that we've added to dplyr 1.2, which is called when any. And this lets you keep the commas. And write it like this. I want to drop rows when any of them are deceased or when the date is less than 2012. So when things are inside of when any, they're combined with or instead of and. Perfect. Awesome. Thank you.

Wife-driven development

Now, another thing that we mentioned way earlier in the chat and Lauren and Raha, I think both had asked about it was, did Libby say wife driven development? And yes, I did say wife driven development. Davis, do you want to talk about that? Yeah. So my wife is kind of like a, she works for a nonprofit and she does like a lot of data analysis work. And she, we both work from home. So she's in the office, like right above mine. And I often will go up there and like look over her shoulder because she uses R and she uses like all the tidyverse stuff and all dplyr and all the packages that we've created. So I would look over her shoulder and she would be like, why does it do this? And I would just like shake my head and be like, gosh, darn it. Like you pointed out like another hole of like something that we need. And I have to go do more work now. I'll be back in like half an hour and I go and figure out like a function that she needs and add it.

So like with filter out in particular, like we aggregated like all of these examples of like where people really needed filter out. But like the most compelling to me was this one right here. My wife has brought this up to me at least three times. So that is called wife-driven development, ladies and gentlemen. That is wife-driven development at its finest. I love it.

So that is called wife-driven development, ladies and gentlemen. That is wife-driven development at its finest.

Okay. I wanted to address something that I see in the chat, which is lots of chatter about like negating things with the bang and mentioning like all of the creative ways that people have learned to circumvent this. We no longer need it, but we're all kind of used to adding that bang to everything. I want to also mention that there is no not in or there wasn't for a long time. There was like percent not percent, but percent not in percent has been or is being added to base R. It was in the releases, the release notes a couple of months ago. And I saw some people who were like, I've literally never needed that. Why would we need it? And then thousands of people who were like rejoicing because they've been adding it to their profile or something for a really long time.

So it's another good example of just because you don't use something doesn't mean that everybody else is not like really, really wishing it existed. I will add. Let me see. What do I have? Do I have a link? Maybe I can find the link to the development news, but I will put just the little like a bit of context in the chat for how it was added in there. I just loved the description on it because the description says provided almost entirely for convenience and code readability from an R DevRel suggestion after many years of private definitions, mostly hidden in packages, including in R's tools package. Amen. We've all been hiding it for a long time.

Community questions: dplyr and Snowflake

OK, so there was another question as well. Let's head back over to Dave's question. David asked, I'm interested in your thoughts on incorporating the power of dplyr into Snowflake pipelines for data transformations. I am coming up on converting some Alteryx workflows into Snowflake without dbt. Is SQL the only option for setting up the staging, marts and tasks? I think you might be looking for dbplyr. Isabella or Davis, would you like to chime in here? Isabella? Oh, yeah. Well, welcome. Your thoughts, Davis, but dbplyr allows you to use dplyr syntax on databases. And I've definitely tried it out with Snowflake databases before. So give it a shot and see if that might be a solution. It does look like we have an official like it does look like dbplyr itself as an official like Snowflake backend.

I'm sure if something doesn't work quite right in there in terms of how things are like translated to SQL via Snowflake, I think we'll be happy to take that issue and try and make it a little bit better for you.

Perfect. All right. Thank you. I just wanted to add an update on Luis's question because Tan found, I think, what we were talking about with the anonymous notation. And it was inside of a vignette for recode and replace values.

I can share my screen really quickly. Aha, you found it. Yep. There is sort of anonymous notation there, or it looks like it in the bottom. It's really just the dot X, I think, is probably what we're talking about. There you go.

Okay. Yep. Awesome. Thank you. Is there a question? Right. Like we're just like wondering what this is kind of thing. Yeah. If you could explain it.

So this is actually something that when I am teaching, this is a super common question that will stop learners in their tracks because it's not intuitive. It's not explained anywhere. It's one of those things like, you know, hey, I'm looking in the help documentation and I keep seeing dot, dot, dot everywhere as an argument. What does that mean? But it's not explained anywhere. It's kind of tacit knowledge.

Yeah. For a long time. So what we're trying to accomplish here is say for every column that starts with score in its name, I want to do something with that column. In this case, I want to recode the values in that column from something to something else. But inside of recode values, we need to reference that column somehow. I need to pass it to recode values. So there wasn't really an easy way to do this in R for a long, long time.

So what we had created inside of like RLANG and tidyverse was this like very compact kind of formula, one-sided formula notation where like you put a formula and on the right-hand side, you get to reference this like dot X here. And all that really is equivalent to is function of dot X recode values. And it just kind of allows you to get away with that without having to put the function up on the front side. And then any column that starts with score kind of gets subbed in for dot X, like whenever you execute the code.

Now, as of like relatively, you know, newish R, you can do this instead. And this is probably like this was probably motivated by like us adding this many years ago. But now directly in R without anything from the tidyverse, you can do these nice little anonymous functions. And this is still exactly the same as putting function here. This little, you know, backslash, forward slash, whatever, whichever one this is. Parentheses X is exactly the same as function X. And now you don't really need the tilde dot X anymore. You can just use this form instead. And we don't care what you use. Like we have no really strong preference, but we've probably started to migrate most of our docs to the new form. Same with the base pipe. Same kind of idea.

same kind of idea perfect thank you louise for asking that question it is a very very good question um that notation is super super confusing it's confusing to me i look it up every single time i use it and thank you to tan for helping us find that example that was really really helpful teamwork in the chat y'all

Deprecation and superseding in the tidyverse

um okay so speaking of things changing and us not wanting you to do something anymore can we talk about with our last little like 20 minutes here can we touch on deprecation because what recode values entering the chat means is that case match is being deprecated and there are some people who are like you will pry case match from my cold dead fingers um but it's a drop-in replacement can we talk a little bit about what deprecation means for recode values and case match um specifically but then also how deprecation decisions are made in general the difference between deprecating and superseding

yeah i think it's helpful to just have this little chart up yeah let's do that this is i assume you guys have the link to this so you can share uh there's a package that we use called life cycle and this kind of lets us manage how our functions in any given package is how they're deprecated like it lets us manage the life cycle stage of that function most of the time the tidyverse lives in this kind of green staple boat bucket here it won't necessarily say stable or anything but if it doesn't say anything special you can just kind of assume it's stable um now whenever we decide that like we made a bad decision or something about something or we want to change something and replace it with something else things will get moved from stable to either deprecated or superseded

now the difference between these two is superseded is like it's frozen in time for forever like you will be able to use this thing for infinity amount of time but we will not make any updates to it we will not add new features to it we will only fix the most critical bug fixes but it will continue to work for forever deprecated is the idea that we will eventually like to remove this it might take us five years but we would eventually like to remove this thing so that's the two ways you can think about what like what these two words mean

now the difference between these two is superseded is like it's frozen in time for forever like you will be able to use this thing for infinity amount of time but we will not make any updates to it we will not add new features to it we will only fix the most critical bug fixes but it will continue to work for forever deprecated is the idea that we will eventually like to remove this it might take us five years but we would eventually like to remove this thing

like the way that we decide like is a function you know being targeted for deprecation or superseded it's a little fuzzy uh you know in tidyr we superseded gather and spread because those existed like from the beginning of tidyr it's been so many years like it's embedded in people's documentation if they're teaching like in their slides and how just how people learn tidyr from the beginning and sometimes with gather and spread so there was no way that we could ever get rid of those but with case match like that was kind of introduced in dplyr 1.1 which is the version you know right before 1.2 it's only been a few years like we don't think that many people actually use it when we're looking at packages uh and it was kind of a really bad name and we have a really direct replacement with recode values so all of those things kind of piled on and said okay it looks like we could probably remove this thing so we've decided to deprecate case match instead so it's a little bit fuzzy in terms of exactly how we decide but that's the general idea

all right how should people think about the the decisions that they need to make in the moment when they see something that says deprecated or superseded or experimental and if you would like an example of an experimental function you can go check out something like nest by i'll stick that in the chat for everybody if you go click that page you will see that it has a little tag that says life cycle tag experimental so if someone sees something that is experimental or deprecated or superseded what decisions should they be making about whether or not they should use that and in what context like should i put this in my production code

yeah i think um you know if you're a die-hard gather and spread person and you used it before it was superseded and stuff and you really don't want to switch i think that's totally fine like if you saw that it's superseded all of a sudden you were like i don't i but i really like it like that's okay but if there's a function that you stumble across and it says that it's already deprecated or it's already superseded like you should probably looking be looking for the the replacement and learn that step like i wouldn't encourage learning the superseded or deprecated function if you've never used it before

um for experimental um i mean we we like feedback on the thing we try not to break too many things like that are experimental we hope to move them into stable but experimental is our way of saying like we aren't quite sure about this yet like we feel pretty good about it uh it seems useful but if this is one of those things in particular where if you have feedback on it or it doesn't feel right we'd really welcome a github issue about that but i think you should feel free to use it

yes and if you are ever on the dplyr documentation um in the tidyverse.org website and you see a tag it says life cycle something looks like a little pill with half one color half the other um you can click on experimental or deprecated or whatever and it's going to link you directly to the definition of what that means which can be really helpful um okay i had a question from tan which was is there an internal rule of thumb on how long something stays experimental like when is nest by going to be non-experimental that's a good question

not really like typically like i would say a minor release which is so you know moving dplyr from 1.1 to 1.2 we moved like the dot by function from like mutate and summarize that was new from experimental to stable so i would say like a minor release now it just depends on the cadence of that package of how often is a minor release released you know sometimes it's for dplyr it was like a couple years for tidyr it might be like every year something like that but i would say roughly maybe a year is about how much time it takes us to kind of figure something out decide it should be moved to stable or not right

Replace values example

all right cool well i wanted to ask you if you could give us one more example with replace values because we talked about recode values um why would we need a replace values and i'm gonna stick a gist in the chat for replace values um that people can follow along if they would like to sure

while we're waiting brent pfeiffer had asked how often do you have these wife-driven coding moments more often than you think yeah i've had quite a few um there's a lot of there's a number of functions in like the clock package which have been motivated by her because she uses that for she does a lot of time series stuff and um yeah like a number of things in dplyr and tidyr as well which have been just originally motivated by some kind of wife-driven development it's so funny i have the clock package is one of those packages where like i have the hex for it but i've never used it and i feel like such a poser i'm like look at me with my hex for a clock i think it's so cute and yet

um yeah so raha had asked what exactly is a gist i had shared a gist what a great question a gist is like a really uh self-contained single file or a couple of files instead of a repo right so you can just click it and you get straight to the text version of whatever it is that you've been linked to so if you click that gist it will pop up and just literally just be a dot r file um but there are things you can do inside of gists like you'll see if you click that that davis commented on it so he's got comments in there it's just a really self-contained way to send somebody a little packet of like here is a snippet of code but i have often sent things to learners that have like a dot r file but then also you can add multiple files so i can add a dot csv so straight from one link they can go like download a csv and grab the code file and run all of it so if you've never seen one you can make one at like gist.github.com if you just go there it'll log you into github and you can create your first gist and you can make them private or public

all right davis we only have about 15 minutes left would you like to give us an example of replace values sure

yeah so we added recode values which is kind of use it's useful for replacing every single value with something else so as we saw over here when we were recoding we replaced every single value in score with something else like one two three four and five all got changed to some other value and in particular the type changed as well like it was a numeric and then it became a character but there's also this like very common scenario where you're just like patching up some column in this case we have some state column and things aren't quite right like we have some missing values we have some unknowns we have some not recorded it would be nice if we could like standardize all these things but we still want it to be a character vector and i still really just want all of the good values to stick around

so in the past like you might have done this kind of patching with like a various like a variety of different tooling like if i wanted to say replace all of the nas with unknown maybe that's my first pass like i might use if else and say like if it's na put unknown otherwise keep state and like that works fine similar with coalesce similar with tidyr replace na all of these little utilities that we've created in the past over the years of different ways to do this they all get the same thing

with replace values it's similar kind of case when style like interface that you've seen before except like it starts with state and you can say whenever there's an na put unknown and then the kind of implication here is but keep everything else the same so the na's become unknown but all of the other values from state stick around and that's like something that you may have done with uh like case when a lot like the similar kind of case when-y thing here is a good old dot default yeah yeah isn't a state put unknown and a good old dot default of state and that wasn't supposed to be there and it's just kind of weird to me that like state the

thing that you care about was the last argument here it's like the default argument it's the last thing you see when really like when you're doing some kind of partial updating like it feels like it should be the first thing that you see and that's what you get out of replace values that like change of intent is really like the really most important thing about replace values that just kind of feels right same problem with if else and then coalesce was kind of nice here but and not as useful because it only works with the nice yeah coalesce is one of those functions that not a lot of people know what it what it does do you want to talk about that a little bit

yeah so coalesce comes from like the sequel world it's like a direct translation uh from the sequel world uh and and all that it does is say it says okay anywhere you see na i want you to give me some other value to replace that with so like state has na's and i want to replace that with unknown so in this case state gets replaced with unknown it's a little bit more complicated than that but like that's the general way that people use coalesce yeah it's one of those that if you're not if you're not coming from sequel you might not ever think to look for it or know that it exists there's all kinds of functions in dplyr that are like that um and what's the question

yeah why is the function called coalesce is that the same term as in sequel yeah we didn't have control over that one right that was a legacy from sequel um i'm going to ignore the fight in the chat over whether or not it's gist or just because uh it's okay you you say it however you want to say it i say gist because of github github gist i assumed it was a play on words um

Package archival and reverse dependency checks

we had another question that was a little further up from russ that had said hey i received a message this morning that one of the packages that i use in my packages is going to be archived next week it's unclear whether or not this is a tidyverse package but some kind of package he says i'm working to fix this but is there a list of packages or functions that are going to be archived so i can get a little bit more lead time if this ever happens again i think that we can talk about the way that um posit with the tidyverse packages approaches this and how they let people know because there is a lead time where they let people know a couple months ahead of time davis can you talk a little bit about the behind the scenes of that yeah so um it's unclear if it's us

that broke this package or not uh from from what he says but if we are going to break your package we typically know because we run reverse dependencies checks on like you know dplyr has like 6 000 packages that depend on it we check the uh tests of all 6 000 of those packages before we send a new version of dplyr to cram and if anything looks like it's going to break because of the changes that we make we go ahead and send the pull request to that maintainer and say like hey it looks like we broke this here is how you fix it and it's just our way of doing some kind of community outreach community goodwill and hopefully making any change we make a little bit easier to swallow uh so like when we do that we give people especially with a big package like dplyr we try to give people around a month to update their package before we push something to cram and then whenever we put something on cram cram will give you another two to four weeks depending on the package to actually fix the thing before there's any risk of like actually getting kicked off so i'm not exactly sure what's going on in this case but that's the general way that we look at things perfect thank you we had a couple of other questions that were function specific

we typically know because we run reverse dependencies checks on like you know dplyr has like 6 000 packages that depend on it we check the uh tests of all 6 000 of those packages before we send a new version of dplyr to cram and if anything looks like it's going to break because of the changes that we make we go ahead and send the pull request to that maintainer and say like hey it looks like we broke this here is how you fix it and it's just our way of doing some kind of community outreach community goodwill and hopefully making any change we make a little bit easier to swallow

Using coalesce across multiple columns

and i figure since we have a few minutes here we can go ahead and talk about some of those one was from daniel that said could coalesce be used on multiple columns

um uh you could certainly use it like with uh i'm not exactly sure what you mean here like we could put uh so one thing you can do is put like another column here instead of unknown and you know if it was the one two three fourth value of state that was in a and i had you know another fall back column that was here it would take the it would take the fourth value of fall back and fill that in for na so that's a way to put two columns in coalesce if you're asking if you want to like run coalesce over multiple columns you can do that with a cross similar to the other uh like example that we showed from github perfect and russ did get back to us and say it was not anything in tidyverse that's going to get archived next week sorry for not being clear about that um yeah you know one of those things about open source development is that a lot of people have different standards for how they do things um and whether or not they are organized and being headed by somebody who is doing project management and making sure that those things happen so i think it's probably different for everybody across the board but that's the way that we do it here at posit

The dot dot dot argument explained

uh there was another thing that i wanted to call out that was i think alex had mentioned somewhere that like hey libby mentioned dot dot dot but then we didn't talk about it um and that is a really really common question that i see about dplyr documentation but really ner documentation is like why do i keep seeing this dot dot dot as an argument option and what does that mean and it really is a placeholder that's saying this function can accept any number of extra arguments um and those are usually arguments that are being passed to like another function or even a function that is inside of that function davis do you have a better and more eloquent way to describe the dot dot dot placeholder that's pretty good i think i think you could think of it like like inside of mutate for example uh the dot dot dot means i can take any number of conditions you can add two columns you can add five columns you can add 10 columns in this particular case it says you can give me any number of columns or expressions that you want to add so typically in an example like this like we use dot to dot to mean like any number of the thing and that's a pretty concise way to say it i think

okay yeah i think it's one of those things where it's not explicitly explained anywhere and i have i have had questions about it for years i am constantly going like man dot dot dot what what functions where should i be looking to see what other arguments i can use with this function right like just give me a menu of options so that i know what i can use because then i'll see somebody's code somewhere where they have um an argument in this function that i didn't know you could use there right and it was just encompassed by the dot dot dot and that feels so frustrating as a user um i don't know if there are any resources anybody knows of to like see what their options are when they're using that but oh thanks ryan as well for linking the dot dot dot section of advanced our book that's really helpful

Wrapping up and announcements

all right if we do not have any more questions for davis about community-driven development or dplyr or behind the scenes stuff i will say we can go ahead and wrap up but i will remind everybody that if you registered for rainbow r i think registration closed yesterday that is happening tomorrow and the next day i will be there i will be doing um a short data science trivia with domi pack you can also um let's see we have call for talks open for use r for a little bit longer i think isabella that's right right the call for talks for posit conf has closed it officially closed closed on the 20th thanks for everybody who submitted um and then also we do not have a data science lab next week so same time same place something else is happening actually if you would like to sign up for it um johnson and johnson is going to be doing a webinar so satish from johnson and johnson will be talking about the admin perspective the it perspective of scaling data science ides specifically for them that's positron as a way to kind of combine everybody onto one platform instead of everybody in the wild wild west kind of doing whatever they they want to do everybody gets their needs satisfied by positron i think it's a really interesting glimpse into the it side of something

um and then oh do we have one more question from alex you want to read that isabella sure uh alex has any chance duckplyr going mainstream and included in dplyr duck plier oh i'm sorry duckplyr did i say duck dplyr twice my bad can you say that one more time so i need to try it yeah any chance duck plier going mainstream and included in dplyr okay that's the order you're asking okay um not directly probably anytime soon has like the native back end for dplyr or anything it still feels pretty good to have people opt in to duckplyr i think it's gotten pretty good at being like a uh one-to-one type replacement and like having a really good fallback mechanism but i don't think it's good enough to just like swap everyone's code out right now with duckplyr as like the underlying back end um so not not quite yet perfect

yeah thank you so much everybody for all of your questions if you have any more questions that you have not thought about yet but you think about them later stick them in the data science live channel and we can talk about them thank you for everybody who hopped in and answered everybody else's questions as well have a fantastic week everybody davis thank you so much for joining us and giving us all the behind the scenes info thank you all no problem all right bye everybody have a good week thank you