Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Lab, everybody. I'm so glad you're here. This is a space where we do live coding, technical topics, screen sharing, and we talk very openly about learning data science topics. We talk about all kinds of things across all the data science topics.

We are joined by a rotating cast of people from either the community or also from inside Posit, where I work, that can teach us, share things with us, and also showcase projects that they've worked on. My name is Libby. I do community stuff here at Posit. I'm joined by Isabella, my co-host. Isabella, would you like to say hello? Yes, hi, everyone. Thanks for joining. Hooray. Isabella, do you have any housekeeping you would like to announce to anybody about anything? I think the biggest thing is Posit Conf is coming up in a few short weeks. It'd be lovely to see you either in Houston or online. I'll share a conference page with everybody to check it out. Perfect. We would love to see you.

Isabella and I will be shepherding your online experience at Posit Conf. If you're joining us virtually, go check out the virtual join options. And if you are going to be there in person, I highly recommend going to Posit Conf in person at least once if you possibly can. It's a wonderful experience. And maybe our illustrious lab manager today might be there in person. I would like to present Rich Iannone. He will be okay. I thought so, but I was like, I might be wrong. Rich Iannone, would you like to introduce yourself, Rich? Yeah, yeah. My name is Rich. I work at Posit. And I work on mostly open source, like R packages, and also more lately, Python packages, or are they called libraries? Maybe a little bit of both. But some of the things I've worked on in the R space are gt, blastula, pointblank, this thing that we're going to look at today. There's a bunch more. In Python, I'm working on some ports from those R packages to Python libraries, like GridTables is essentially gt in Python. There's pointblank also in Python. I'll touch that a little bit today. I'm not going to go through a code, but I'll just mention it and show you the website. And something called GreatDocs. Basically, it's a way to, it's like pkgdown for Python, essentially. You know, pkgdown. It's basically just a site generator for your Python libraries. But nice. Hooray, we love pkgdown. Yeah. All right. Fantastic.

So now that you know Rich has his hands in a whole bunch of packages that you've probably used and love, or maybe you haven't used them and you're going to go explore gt, GreatDocs, GreatTables for Python, blastula. blastula is emailing in R. That is all you need to know. It is amazing. It gives you all kinds of functionality. Go check it out. But today, we're going to be talking about pointblank. And pointblank is all about data management and data validation in R with so many quality of life features for you that if you are used to using something that's very manual, like the validate package, you will absolutely be blown away by what's possible with pointblank. So with that said, Rich, I would love it if you could share your screen and let's kick off with some quick discussion about data validation.

Overview of pointblank and the films dataset

Yes. First, I'm going to show you where stuff is just real fast. Just so you don't have to capture the screen. All the stuff I'm going to show you inside Positron is actually on my presentations repo. So just look for my name and presentations. And it's actually in this folder right here, dslab, with the current date today. There's not much in here, but there's some stuff in here. It's mostly empty because we're going to muddle through some pointblank code, so it's not all filled out. But the main things are there. And the thing we're going to focus on is this package, pointblank. And it's been around for a very long time. And the whole idea is that you have data and you want to keep track of the quality of that data. So you're creating checks. Maybe new data comes in and it sort of appends or your data gets updated. You can keep on top of whether the quality is maintained. Or maybe you just have a problematic data set and you want to sort of cut down on the errors and fix it. Or maybe you get a lot of third party data. Or also, you just have a notebook and you want to do periodic checks during your transformations, like in between, to make sure things are nice and good. There's no surprises.

So check this out. But that's a later thing. A lot of the stuff that we'll work on today, and it will be just basically like the most simplest example, will be just right here in Positron. And the reference right here. This is just to jog my own memory. Because even though I work on this package, I haven't really looked at it in a year. No, actually, no. That's a bit of a lie. But it sounds dramatic. But it's actually kind of fun. Because, like, I don't know half the stuff. Like, I don't remember this stuff. I didn't want to prepare either because I want to keep it, like, you know, like, some crowd participation happening here. So if you want to shout out something. This is very real. You go back to your work a year later and you're like, what did I do? And why did I do it? Yeah, exactly. It'll just be like, I'm a learner too. So that's the whole guise of this thing.

So let me tell you about how this is going to go. I'm going to, like, work through this QMD right here in Positron. And I brought a data set, actually. So the data set is let me see if it works. Run. There we go. Okay. So it's working. So I created a data set just for this cool event. And it's called Films. And basically, it's just, like, stuff I watched. And I tried to, like, get down every movie I watched inside of an app called Letterboxd. It's like a site as well. And, you know, I tried to, like, you know, remember every single film I watched. I think I failed because it's a major undercount. But I tried. And I rated every film too just as, you know, to be completest, you know, to have a full data set. And I'm glad I did this because now we have a data set for this event and this lab. And I'm going to load that in right now so we can have a look at it. So Films. It just loaded. I just used read CSV from readr. It's one of my favorite functions for reading in a CSV. Now let's take a look at the tibble. Okay. So real super small. It's all here crammed in this little console. Not the best way to look at it. So what I'll do instead is because we're in Positron, we can use the view function. So let's use that on Films. Ah, that's a lot better. I'm going to make this a little bit bigger. Here we go. And these are some of the movies I watched. And if you scroll a little bit, we see a lot of data here. And the cool thing is what I did to get this data is basically just, like, you know, sit one afternoon, maybe two afternoons on the Letterboxd app, read a bunch of films. And then I just grabbed the data and assembled the data set. If you log in, you can just grab the CSVs and sort of assemble it. It's actually not as straightforward as it should be.

But it's all there. Potential's there. But I have this data set now. And another secret about this data set, I tried to make it a little bit, like, bad just for this event because this is all about data validation. And I sort of corrupted it. I changed things and I made some things, like, not right. So we're going to try to find those things. And we can't do it just by looking at the data. Like, this is impossible. Like, how many rows do we have? 1,000? Even that's too many. I would love to know, yeah. How many movies are in here? And also, what is your scale? Is it 1 to 5? It's 1 to 5. Yeah, I believe that's how it works in Letterboxd. Yeah, maybe even half, but I think it's, I think they allow half. But I think you have to at least, you can't have zero. Okay, 1 to 5. It's not like a detail, but this is super important to know. Over 1,000 movies? You've watched over 1,000 movies? Yeah, this, well, this might be, like, this is, like, in my entire life. So, you know, like, I had to scan, you know, like, like, I had to, like, think, jump around. It was pretty much chaotic. Sort of, like, jumping around, different directors, you know, like, oh, I saw, I saw this. Yes. And I, and I think I got, like, most of it, but there's surely stuff that, you know, I've seen that's not in here. So, you know, I do my best.

One more thing that I want to get out there just for a primer, especially for anybody watching this on YouTube later, is a definition of data validation and why it's needed. You might get data from all kinds of different places, and one scenario is maybe you have a dashboard. Your dashboard is a composition of, like, five different regions. You're getting data that comes in every week or every day from those different regions, and it's all in a format that you would expect, right? Everything's coming in in the right way. It's feeding into your dashboard, and then something changes in one of the regions or a couple of the regions. Maybe there's a software update in one region and not another, and your data is coming in slightly different. Maybe it's not different enough to break things, so things are still coming through, but the data in your dashboard itself is now wrong. How would you know that it's wrong?

How would you know that anything had changed? This is why data validation can be really important, so that you make sure that you are on top of everything coming in. Is it the right type? Is it within the correct range? Is it in the right format that you expect for your code to work? All of that stuff. Data management is a broader topic, but that is at least the gist of data validation. Yeah, and Libby, I've been there. I've seen all those issues. I dealt with, like, when I worked in analytics, third-party data would always come in. I mean, it would come in, which is great. Sometimes it wouldn't come in. I think you got to check, but sometimes the big batches of it would just be wrong, or a process would change, and people wouldn't know about the third-party data enough to integrate it properly into the table. Lots of issues. Oh, my God. That's why this package exists, because I tried to deal with it. Because your own woes. Yeah, exactly. Yeah, yeah. But now we have it.

That's why this package exists, because I tried to deal with it.

Exploring the dataset with scan_data

So you dirtied up this dataset just enough that it could simulate things coming in wrong, missing, whatever. Okay, cool. Yeah, and here's the thing. Some of these things are obvious. There's a difference between plausible sort of things, like implausible, and subtle sort of small things. Say, for instance, a movie could be, the year of release could be, like, 1825. Sounds wrong. Way before the advent of films, which was like 1880s or 1890s. Or there might be a movie that has an actual release date of 2002, but was actually listed as 1998. It's like, how do you know that? I mean, that's tough, but we're going to try our best with this dataset here. So I got errors like that.

Okay. So we looked at view. And I always recommend just taking a look at the data. Another one that people reach for a lot in R is, like, summary, which is great. I sort of see it here. I'm going to scroll this up a little bit. It's nice. It's not as good as view, though, because you can't see what's missing, like, how many missings you have in character columns. You have NAs for other things, but not for character. So it's not bad, but I think we have better these days. And actually, pointblank has a don't thing. It's got a function called scan data. And rather than just talk about it, I'll just show you what it is. And one thing I did is, it's a little bit slow in some aspects. So basically, it creates an HTML report of your data. And there's an overview. That's what O is. And then V here is, like, variables. And S is the summary. So you can choose what parts you want in the report. So I'm just limiting the entire thing and just choosing these parts. So I'm just going to run this. So it's going to do that right now. It finished it. I'm going to scroll over here and open this up. So basically, this is, like, an overview of the dataset. So it shows you, like, just some top level things, like, you know, like the dimensions and how many NAs you have in total across all cells. Duplicate rows, even. And this is the really cool thing, like, variables. Like, film titles is, like, one variable. We can actually click this and see cool little things. We see things like duplicates, like, right here. And also troubling NAs for, like, the film. Like, if you don't have that, that's a bad row. You know, like, the thing that you actually want to, like, you know, describe is, like, missing the name.

I think we caught right away just by doing some simple data inspection. We see a lot of, like, titles which have, like, you know, duplicates. That's fine. There's, like, remakes. And even, like, the Avengers has two different films entirely. So, yeah, we have that. And we also have a year. This is actually really good to see. Because we see, like, you know, like, the stats here. Like, the mean is not very important. But the min and the max are very important. So we see right away we have a minimum of, like, 53. Sounds kind of bad. We want to have a full year. Not, like, an apostrophe 53. We just want the whole year. And a max of, like, 2108, which does not look good for a year. So we're already seeing stuff that's kind of bad here. We've seen some NAs across the board. We have some here in this URI for, like, the Letterboxd to get to, like, the info page for that movie. We see some NAs. So that's already kind of bad. So in our validation, we're going to, like, you know, firm that up and put in a check for NAs. Pretty much everywhere. And the film ratings, we just talked about that. It should go from, like, I think 0.5. Like, this is what's allowed in terms of the form to 5 inclusive. It goes in increments of 0.5 as well. And we see here a minimum of minus 3. Bad. Maximum of 8. Also bad. So these are problems. And the cool thing is we can see, you know, like, more stats here. These are useful or not useful, depending on what the measure is. But I think just looking at min and max are very, very useful. And right here, we see director and we have some NAs there as well.

I think we learned a lot just by looking at this. First of all, this thing exists for you to just, you know, check out a new dataset quickly. And also we found some, like, things we can validate right away. And we know we're going to catch some errors just by knowing we have NAs and ranges which are have values which are improbable. I also got to say, I love the toggle details button. That's so nice. Yeah, it's kind of nice. Like, sometimes you want the details, sometimes you don't. It's there when you need them.

Yeah. But, like, defaulting to it closed means I can go through this much more quickly when it pops up on my screen and I can only open it to be a bigger when I need it to be.

Well, while we are here, let's ask some questions that are fun without digging in yet. Noor wants to know how you define a five. So, for our own edification, what are your film requirements for reading something a five? Yeah, I don't want to see it again. I don't want to be in that world again, like, a year from now. It's actually, like, it's weird. Like, sometimes I get a little bit obsessed. Like, I see a movie, like, sometimes, like, three times in a two-month period or even a one-month space. It's like, wow, I got to see this again. And it's like, maybe that's a five. Usually that's a five. Like, when I want to see it again, like, a strong impulse. I usually rate pretty high. Like, most films I rate or movies are, get, like, a 3.5 just, like, automatically. Because, you know, it's a movie. A movie is awesome. And, you know, I think unless you're, like, really trying to make me mad, it won't go too far below 3.5.

Okay, good to know. It's nice from a data standpoint to know what we're working with here. Thank you, Noor. Excellent question. And then our second fun one is from Trey. Trey says, hey, what Positron theme is Rich using? Do you know the name of your Positron theme? I forgot. It's a good one, though.

Command Shift P, theme. I bet it'll tell you what it is just by, by Command Shift P theming. Do you have one? Oh, no, it doesn't. Maybe I don't even have one. I'm just using the stock ones. Actually, it would tell me right here in settings. Oh, okay, good. Themes. Nothing. What? No, dang it. Anybody have a really quick way of, like, figuring out which theme you're on? I know exactly which theme I am. It was definitely a set it and forget it thing. I really, really love it and talk about it all the time. Okay, follow up. I'll find out. We'll figure out. Okay, Trey, we'll let you know.

This seems great. It's, like, right on the money. It's, like, it's nice. It's, like, not too dark. It's not too light. We're all going to learn. Isabella and I are probably, like, both, like, feverishly Googling how to very quickly find this in our themes. At this point, it was, like, here, like, in, like, settings and theme, but I don't know. Failed in that regard.

Overview of the films dataset

But anyways, okay, this data, let me, we already got intuition here. It's not that big, like, in terms of columns. It's that way because, you know, I don't want a huge data set here in this lab. So, basically, film title, year, these are self-explanatory. The URI, it's basically just for getting to, like, the, you know, to Letterboxd film page. It's a shortened link, but, you know, no big deal. Film rating, we went over that. It's from, I believe, 0.5. You can't do zero to all the way to five in 0.5 increments. Director, this is a cool thing. It's a first name, last name, and there's multiple directors, and you said sometimes it's just comma separated, just like this. So, it might be useful to check that if that's a, you know, like, a thing you want to hold to. Not always important. As long as it has a director. Usually all films do. But there's always exceptions. Data is very surprising.

The pointblank validation workflow

Okay. So, pointblank. Let's get to the thing we're going to do. So, the workflow is basically three components when you define a validation. You've got to create an agent, and this is way before AI LLM stuff came out. It's a little unfortunately named right now. But agent is just, yeah, nothing. Big coincidence. There's no AI here. Yeah, it's not. The idea, I think, a long time ago, the theme of this package, pointblank was like, you know, we have an agent, it's going to interrogate the data, you know, really torture it, really violent stuff you're doing with the data.

Wait, that begs the question that Brent just asked, which is, is pointblank named after the movie? Oh, the Lee Marvin. Yeah. No, it's not. But great movie. Great movie, by the way. Okay. Yeah. Yeah. It's not a bad name. Like, it's pretty good as far as names go. But, yeah, that's, that's, agent is basically nothing to do with AI. It's basically just like the entry point. And you can define what the data actually is in this, like, one function. And then after that, you go to, like, validation functions. So, we have a ton to choose from. This is basically, like, each one has a different interface. It's kind of like if you just test that, there's, like, a bunch of those, like, testing functions, and, like, they all have, you know, their own little thing. Like, this value is above this, this value is below this, this value is below this, it matches this, so on and so forth. And then we end off with interrogate. The reason we sort of, like, explicitly say we want to actually interrogate the data now is, like, you can leave it out and just look at the plan, you know, and not touch your data. Because your data may be, it may take a while to validate. It may be in a, you know, like, in a data warehouse. It may, you know, be remote. You just want to, like, you know, like, not do it yet. You must, must see your plan. So, yeah, this is basically like a validation sandwich. You got, like, you know, the entry point, you got the final thing, and then you have all the stuff in between, which is the actual checks. And they run in the order that you define them.

You got, like, you know, the entry point, you got the final thing, and then you have all the stuff in between, which is the actual checks. And they run in the order that you define them.

Yeah. While we are on this point, Hubert asked, how is the speed of scan data on larger datasets? Like, if you have a million rows, but an okay number of columns. Like, not super wide, but super long. It's not too bad, as long as you adhere to, like, not showing all the sections. I'm going to rework this. Some of them are pretty extra, where, like, they create these plots, like, these matrix plots, and that takes up a lot of time. That actually, like, takes, like, sometimes, like, for a huge dataset, like, lots of columns, like, a few minutes. So, I may actually change the defaults, so it just gives you this, because this is, like, pretty much all you need, and the other stuff is just kind of, like, bonus stuff that's, you know, like, if you need it, there you go. But it's pretty fast. Some of the slowness comes to do with, like, just, you know, like, page generation, using HTML tools for reasons. But it's not too bad, because we're basically just using simple queries, and dplyr or dbplyr is the case, maybe, to get the stats.

Yeah. Yeah. Okay. So, again, this part right here in between is, like, the part that takes the longest, and requires the most care. It's basically what you're actually doing to make those checks. And they're all documented in here. So, if you go to this page, I'm going to show you that page real quick. It's basically this right here. Package index, it gives you, like, sort of, like, the lowdown on, like, you know, like, what all the different functions are. And I'll go through them, and we'll just intuitively understand what they are. But these ones that begin with call vals, we're actually just checking individual column values in a single column, so that we see that sort of pattern almost all the way down. And then we look at then we have ones which are, like, rows distinct, rows complete. We're actually checking at row data entirely. And then there's ones for, like, you know, like, type checks right here, like, call is. And then there's some special ones, which, you know, which are, you know, not fitting these patterns. But for the most part, we'll be checking individual column values, because that's kind of, like, you know, like, where you find the most errors, like, in individual, like, data values in a column.

Creating an agent and running validations

So, let's actually do this. Okay. So, let's start. So, the first thing we got to do is basically just follow this. I'm going to actually just do this sort of setup right here. Agent. And then basically just copy down what I have. Create agent. And autocomplete is really just nice. So, table is going to be films. And then I'm going to put just agent here at the bottom. So, I'm going to run this. I'll just, like, see what's going on. I'm just going to run this right now, see what happens. Okay. Great. Okay. So, the first thing you see is, okay, this is actually kind of sweet. Because it runs even with this. Okay. We see, like, this point blank validation plan. This is actually the report. We don't have interrogate at the end, and we don't have any steps. So, it's pretty much, like, an empty view. But this is where you're going to be filling in. This is where you're going to be seeing, like, where the, you know, issues arise with your data set. So, obviously, this is a, you know, like, very beginning. So, we're going to put in, like, some validation steps now. So, now, what's the first thing we should check? I guess for, like, missing values, right? Because we saw we have tons of them, and we want to know to what extent we have them and where they are. So, we can do call vals. I'm going to scroll down. There's lots of them. Call vals, not null.

Okay. So, this is, like, the assertion. We're saying that column values are not null. Of course, we know the null values, but we're going to find out where they are. First argument is usually columns. I'm just going to put them in so we know exactly what we're doing here. We actually want to check all the columns. It would be rather tedious to just say, like, you know, like, to list all the different columns. So, we can actually do everything. We're going to test every single one of the columns for null values. Okay. I'm not going to put an interrogate. I'm just going to run this, and I'll show you what it looks like now on here. Great. Okay. So, we see nothing here. Nothing interrogated. So, we don't have results on the right side. But we have from, like, one line, we're checking, we're making five steps and checking each of these columns. We didn't have to know the names of the columns because we put in everything. We have a selector right there, which is nice because I already forgot the names, the exact names for these columns. I mean, I can look, but this is actually super convenient. So, I'm actually going to run this right now, and we can always, like, add more steps. So, it's going to be interrogate. Great. So, I'm going to run this cell. Great. Now, it's actually running, and what we see now is some results on the right side. I'm going to scroll just a little bit more so we can see, like, the whole table here. There we go. There we go. Okay. So, let's actually, like, take a look at this and unpack what the results show us. So, we see here, like, we'll just look at the first step right here, and if you hover over it, it actually tells you, like, in plain English what you're actually looking for. It says here expect that all values in film title, that column, should not be null. Or should not be null. Yes, that's right. And we see that there's a thousand units, and units are basically just, like, rows and, like, you know, individual cells. So, what we do here is we're actually saying, like, one check is actually checking somewhere around a thousand different values, right? And then we see here pass and fail. These are super important. We have around a thousand different test units passing, but four are failing. So, a very small amount, like, a very small percentage. It's not 1%. It's, like, rounded up to that. It's basically, like, almost like needle in a haystack type stuff here we're looking at.

Right. I have to jump in really quickly. While we were talking about a check for not null, someone had asked, I think Connor had asked, is null in pointblank the same as ours in A? It is, yeah. The terminology is like this. Yeah, I'll explain why I went with null. It's because we also can check database tables here. And it would be maybe a little bit weird saying, like, you know, we're checking NAs in a database table. I mean, both are weird. But we're checking, essentially, missing values. And not including empty strings, though, is a small detail. We're essentially checking NA or null, depending on what kind of frame we're looking at.

Yeah. So, we have some failures. So, these are absolute values here on the top. Like, real raw numbers of, like, failing test units. And below is, like, a proportion of failing. So, 1% rounded up, like, way rounded up. It's nearly 0%. And then we have these little buttons here. They don't actually work in Positron. They're actually, like, okay, these are because this whole thing is a communication sort of medium. You can create this report, show someone else that doesn't know about pointblank, doesn't know about R. And then we get a sense of, like, you know, like, stuff is failing in our table. Like, let's do something about it. I can actually put this in a browser. And now if you click this button, it'll actually download a CSV. And I think CSV is open in the weirdest place.

Should I open it? I'm going to open it. If you want to. I'm going to open it. Let's see what happens. Okay. There's a CSV here. It's opening in numbers. All right. Here we go. That's fine. So, basically, it's showing, like, the different values in the first column, which are NAs right here. And we see, like, this is film title. And we see it right here. These are the rows which have NAs in film title. And it's unfortunate. So, now we don't know what this film by Christopher Kane really is. We know it was, you know, made in 1988. But we don't even know the film. So, this is kind of cool. So, it shows us failing rows, essentially. And I'm just going to get rid of this. All these CSV buttons will do that. Will give you, like, small extracts of, like, failing data. And you can pass those to whomever, you know, is concerned with, you know, like, fixing the table. Could be a data engineer. Could be someone else. Like, a database admin. You have this sort of quick way of getting that up. Of course, we do that through the pointblank API as well. But this is just sort of like a quick and dirty way of doing that.

All right. Cool. I have one question from Renato. And also, time check. We're at the halfway mark. Just thought I would let everybody know. Renato had asked, why do the null stuff instead of is.na or any other NA affordances? And I thought it might just be checking is.na under the hood and just letting you know the results of that. Yeah. Well, it's tricky. Like, we do other checks, too. We're mixing in, like, you know, like, say, for instance, like, greater than checks. And some of those columns may have, you know, NAs in them. And we have to have a way to sort of separate between, like, you know, NAs and, like, you know, the actual numbers, which are, you know, like, not, you know, in the range or whatever. And it's you could, of course, like, have an is.na check. But then it doesn't scale, I guess, you know, if you want to use this one function and you change your data source from, like, say, being, like, an R data frame to, like, the database version of that, which could be, like, a MySQL database, for instance. So, it's just, like, nicely scales or, you know, you write it once and you don't have to worry about the fine details of, like, you know, what is NA.

All right, cool. Thank you. Well, on the topic really quickly of your data can be a CSV or a data object in R or it can be data from a database. Yeah. I know, I think behind the scenes that you, when you are querying a database, this is happening on the database itself and you are not actually processing in your system, so you don't have to pull the data in. So, Lizette had asked for parquet files, do we need to collect, like, the function collect before using the pointblank functions? I think this is a good place to throw that in. Okay, yeah. Right now for parquet, um, I don't believe you do, because you're actually reading those parquets into, like, a data frame, right? You're typically going to read that into, like, DuckDB or you're not actually reading on the parquet itself, you're actually going to read that first into, like, either, like, DuckDB or dplyr. And in that case, for sure, you're not pulling in the data,

but you're doing a first initial read, for sure, before that. Yeah. I hope that that helps, Lizette.

Alright, um, I feel like one more question while we're here, because we were talking about the buttons. Yeah. Grant had asked, has your use of pointblank changed with the Positron viewer being more capable than the RStudio or past Positron versions?

Well, usually what I do with these reports, I don't really, like, use them within, like, the viewer here. It's always going to be part of, like, a core document if they're sent out at all, and, you know, like, published. And in that case, I'm not really using Positron so much. I'm using it as a quick view. But I will say like, compared to like VS Code, this Positron viewer is much better. Like VS Code will do a bunch of like, not nice things to like these outputs. I'm not sure why it does, but it just does. It tends to inject its own like, you know, like table stuff, like CSS. Whereas Positron does not do that. So I'm thankful for Positron in not, you know, wrecking the outputs. I know VS Code is trying to be helpful, but sometimes it's not.

Sometimes it's not, that happens. Yeah.

Adding range validation checks

Okay, I'm going to keep going. We're going to add one or two more checks and- Let's do it. I think we should check for the readings. Because I've got that suspicion. I mean, well-founded that they're out of range. Okay, so basically call vals. And I love this about autocomplete. You can just like see, you have all your pointblank functions here. You can just like choose one that looks good. In this case, I'm on between. Because we want to check, you know, like our readings between two bounded values. And it's going to be a yes for that one. So I'm going to put in the right column name. In this case, it will be film reading. So I'm going to type in the argument. I'm just going to put in film reading right here. And then it's going to be left and right.

Again, not going to put in the arguments, though I could, but these are the next two arguments, left and right. So maybe it's 0.5 and five. And there's more arguments too. Inclusive being, you know, C true true, means that the bounds are including 0.5 and five, not less than, like just shy of the bound. So that's actually good because we want to include five and we want to include 0.5. So that's the default. Let's run that and see what happens. Okay, run cell. There we go. So now we have it added as step six.

So these numbers are really important because if you want to actually see the data, you'd have to refer to these test numbers. So I'm going to do that right now. I'm going to go to pointblank, get data extracts. And then I'll put an agent right here as where I'm getting the data from. And then I, I will be the number six in this case. So there'll be all the readings which are out of bounds. So I'm going to run this and we see right here inside, and we're still as a, there we are. So we see here, we have some readings which are just either NA right here, eight, pangd2 got a really high rating. Well, you know, kind of did a limit break of, you know, like eight. A hotshots maybe got NA. Somebody's got negatives, which are not great. So some data corruption happened. That's my doing. So basically we can find them out. We can find the exact rows, which is quite nice.

Great. So again, here we have a CSV button as before, because if there's any errors, anything here in fail, anything above zero, we're going to have this button appear. Great. Okay. I think there's a few more.

One other one would be, let's see here, a year. Okay. So let's, let's do that one. So same thing, same thing, call vals between. Oops, we have two pipes there on 123. No. Okay. That's way too many pipes. Yeah. There we are. Thanks. In this case, it'll be just a year. And I think I want to choose like something that's like, like plausible. The left one being 1900 and the right one being 2026. Okay. And then one more pipe to close. I'm going to run that cell. And then we see here another call vals between. This case is for the year column. We see here it's between the ranges 1900, 2026. And we see there's 15 items, which are not between that range. Uh-oh, illegal values. Exactly, exactly. And again, here, if you use this, this thing here, get data extracts, we can, through the pointblank API, just see the data that is essentially got wrong years. I bet you a lot of them will be NAs. Yeah. A lot of the NAs here, some films in 1800s, like Pector here, is like 1898 and not 1998. That's totally wrong. Yeah. Speed ratio is like, it's like set in the future, but it's actually like, you know, like the years in the future of a publication date. Totally wrong.

So good to see, good to see. It's, we're catching, we're catching the errors. What if, we have a question here. What if you don't specify that I in get data extracts? Oh. Does it return only the rows that field all checks? Well, you know what? Instead of just answering that, I'll just try it. Let's do it. It does, it does. Live coding, all right, yay. Yeah, basically it returns a list of all the problems. Good intuition, Garrett. And it's keyed by like the step, which is actually super nice. So you can totally use this and run with it. You see right here, we just get like, you know, the first one is like the not null. If you want like this one right here, call that's between, which has to do with the film rating. Let's just go to list item five. Yeah. If your data is pretty small, this is actually a nice little like summary. Yeah, this is not bad. If your data is huge though, this could be- Yeah, you might get tons and tons of errors. It might overflow. Yeah. It would be way too much. But I really like that. Yeah. Great question, Garrett. Excellent question.

Warning, stop, and notify thresholds

How are we doing on time here? Cause I want to get to like- We have 22 minutes left. Oh, that's a lot. And I wanted to touch on a question that Zach asked. I did answer it in the chat, but it needs more explanation and you might get there. And that is, we have the WSN, which is warn, stop, notify. He was like, what are the WSN? I think Rich might get there. I can tell you right now. So basically we have like, you know, these numbers here and, you know, they mean something, but they may not mean something until you ascribe a meaning to them and you show it on report. So basically what you can do is you can provide failure thresholds. Like just basically like if 5%, for instance, 5% of your data is bad, that may elicit a warning. And you might just create that directive to like say, if this goes above 0.05, let's mark it off here in the W column with a, you know, like a marker. And then we may have three of those levels. So we just have three slots available. So warn, stop, notify. So you may have like, you know, increasingly severe, you know, thresholds. Like this could be 5%, 10%, you know, like 20% or something like that, whatever you want. These are like a little buckets for you to say like, oh, it's getting really bad. You may just have like one error might be a warning. And more than that is, you know, like is stop. You may not use this one. So the kind of like semantic sort of like little buckets for you to say like, okay, let's put a stoplight here. Let's, for the person who's seen this report, let's focus on, you know, like the rows that have these, you know, these indicators. Yeah.

Like just basically like if 5%, for instance, 5% of your data is bad, that may elicit a warning. And you might just create that directive to like say, if this goes above 0.05, let's mark it off here in the W column with a, you know, like a marker.

So it's kind of like, you can set that also. I'll just show you how that works through. One sec here. Zoom is getting in the way. Let me just go to. Zoom's always in the way, isn't it? Nobody can see it, but on Richie's screen, there's like this bar in the way. Yeah. So action levels is how you set those things. And you put that, and there's an example right here inside of createAgent. You would just put, as part of the actions argument, you would use this function right here. And then what you get in the end is, although it doesn't show the report, it would say like, it would show you those indicators. And also give you some metadata when you look at the object, whether things actually tripped those levels. Yeah. I hope that answers it.

Custom expression validation

Let me get to like one more really good validation function. Maybe two, if we have time. So we have ones where you could like use an expression of your own, and it's called callValsExper, like this. And this one's like a little more difficult to use. You have to start off with this argument, and then you have to wrap it with the exper function. It's like a helper function. But then once you're there, within that you can do things like, you can check whether pretty much anything you want. You're kind of like customizing your own check. So in one case, in the case I want to look at is that the ratings are separated by 0.5s. Like we have, you know, like half, one, one and a half, two, and so on. So we can do that. And this is the thing I always forget, but I know it's used also in like years. If you're looking at like, you know, like every five years, every 10 years, you basically use this like module. Wrong one. You want to use this, and then use the increment right here. And then if there's no remainder, then it's correct. Like everything is in terms of 0.5s.

And another cool thing I'm gonna show you is that, so it just checks every single value within that film reading column, because we actually specify here in the expression. We also want to make sure we're just, we're not looking at NAs. So in this case, we're going to pass the NAs like this. So in this case, we're saying like, you know, NA won't be a part of this right here. We were not checking any value guesses. It's going to be a real value that's not NA that's being checked. We're sort of like a separation of concerns. Okay, so let's add the pipe here and let's take a look at this in the end. This will be like the step eight. Great, okay. So let's take a look at this more in full. We see here that we have six values, which are numeric, not nulls, that fail. Okay, so now I'm really curious, like what values we have, which are 9.5 increments. So let's actually do that. Get to extracts agent, then I, and it's gonna be number eight. So pull this up. Oh, okay, here, I see it right now. We have a film reading of 4.75, 4.25, another 4.25. Yeah, basically you should not be, these are not right. Illegal, red card, okay. Yes, yes, these are bad values. Yeah, we got to fix these like somehow, but now we know, right? Like before we wouldn't know, but we knew the rule, right? The rule is like, you know, we have this range and there must be these values within that range.

I love your little symbols too. Like their little range call between or the little lambda. Yeah, some of them I'm like, wow, I don't know what that Greek symbol is at all. Oh, the lambda, yeah, yeah. I know what lambda is, but I don't know what the W symbol, where they cross through it. My brain was like, which Greek letter is this? I had to really research these. Yeah, apparently like it's omega. And like, this is like rare knowledge. I just found on some dusty corner of the internet. It's like omega is kind of like, you know, how you say if you have nulls or missing values. Maybe it's a math thing, not so much a database thing, but that's the only thing I could find. Okay, lowercase omega. All right, good to know, learning new things every day.

Draft validation: auto-generating a workflow

Yeah, it's fun too. Also, you know, sometimes you just want to draw things instead of like, you know, write code. So that's that. Okay, one thing I want to show you also, this is like a ton of data validation. I feel like our data is, we've elucidated tons of little issues. With this data set. And now it's my job to fix that data set. Or, you know, if this were a real thing, you know, like that, you know, my job to hit it on, I would fix this data set in whichever way I could. But I want to show you another method for getting validations. We can use something called draft validation. It's a function. And I'm just going to write out draft validation. And you just point towards your table. This case, it is films. You can give it a bunch of other names, a bunch of other arguments. Might as well put table name in there. This is just for the report, I think. I'll just call it films again. And the key thing is you have to have file name because it wants to write an R file to disk. And basically, what's this going to do? It's going to create like a validation workflow for you just based on the data itself. It's going to fit your data to like validations at work. So I'll just call it like test.r. I think it's going to go to like the base directory. We're going to find this. So I'm going to hit run. Okay, so it says like the pointblank.r file has been written. Okay, I'm going to try to find that. Here it is, test right here. I believe that's the one. Yeah, this is it. This is it.

Okay. So basically what it did here is it doesn't, it creates like the whole agent workflow. Puts the name of the dataset in. Even put some like thresholds in for you just as a way of, it's basically just templating. It gives you like a label even. And it gives you like line by line validation steps. Like is this column a character? Does this column have integer values? Are they between? We see right here that it just fit on the data. So this is wrong. But it's actually kind of cool. You can just see this and go, this is totally wrong. I just changed this. This is actually this. Basically gives you a starting spot. Starting point, exactly, exactly. Yeah, it's assuming like the data is perfect. So it's basically fitting on your data, but it's actually fine because you can elucidate all sorts of things. We can, you know, remove some of these. We can like change them. But it's a really great starting point. Like right here, for instance, this is totally wrong. So I'm just changing that to this. But it's cool that it gives you comments, which are sort of like plain English. It's telling you exactly what the step is trying to like check for. And then it's like, great. I got all these steps written pretty quickly based on a dataset. And it just scales with the size of your dataset too, right? Imagine typing, you know, looking at your dataset and typing all these things. And whereas you can just start from here and reduce it, change things, tweak things. This is amazing. And I've already forgot the name of the function. Can you scroll up just a little bit? Oh, yeah, draft validation. Draft validation, okay. Yeah, yeah. Amazing. Thank you.

And it just scales with the size of your dataset too, right? Imagine typing, you know, looking at your dataset and typing all these things. And whereas you can just start from here and reduce it, change things, tweak things.

I'm gonna try to stick a little, a link in there to draft validation.

Oh my God, there's some more. Okay, there's a bit more. One more thing. We have 12 minutes left. 12 minutes left. I can get it in, don't worry.

Splitting data with getSunderedData

Okay, so sometimes you have like a dataset and you do your validations on it. And you find that some of the rows are bad. Like we saw some rows that didn't even have like the name of the movie on it. It's like, those are obviously, I can't use those in, you know, what I'm gonna do with the dataset. Pretend like I'm gonna do something with the dataset.

So what we can do is use this function called getSunderedData. I'm gonna just put that in right here. Okay, let's take a look at that. So you got your data. GetSundered. Yeah, it's like splitting it apart. Like, you know, like torn asunder. You know, that sort of thing.

Yeah, that's what it's trying to do here. We got agents and what we're doing here is we're splitting it into like two different pieces. And one piece of the dataset has all the rows which passed validation entirely. And we have another set, which is, you know, like the rest of the rows, which failed in some way, like some value in a row has failed.

So we're gonna sequester those rows to like the fail piece. But by default, it gives you the good piece.

So let's actually run this. And really, okay, so if you look down here, it just really passes through your table. So now we have like 10, 14 rows.

So let's compare this against, you know, just like films. I'm just gonna erase this for a sec, which has 1055 rows. Okay, so what we've done here is basically we have like, we've applied sort of like a filter to like our dataset and are just filtering the good rows. We can do the, you know, the thing where we just want to see the bad rows and that'd be type equals fail.

And this would be, you know, of course, a very small number of rows. So here we see, we have 41 rows, but it keeps everything the same. Like, you know, it's all the columns are there. And if you were to, you know, like add these like together, like the row counts, it would equal up to like the original dataset. So nothing is lost in this. You're basically just splitting your dataset based on your validation.

So this is good. So like, you know, my job would be like, oh yeah, I got these 41 rows. You know, I gotta rate, I gotta rate this movie or, you know, I gotta find a year for Buffalo 66. You know, what year is that in? I gotta find out.

Yeah, yeah. So it gets under data. Useful at times when you have data, you know you have bad data, but you want to use some of it, you know, and you probably can't use the bad data, but you're fine with that. You're fine with that. Or you want to know what the bad data is so you can fix those problems.

And it's pretty flexible as well. You can do things like, you can also like, you know, instead of like, you can put the entire dataset and put a flag column in instead and say like, you know, and choose what the value is. So you can have the entire dataset just sort of decorated with a new column and it has like, you know, yes or no, or one or zero. It's sort of like, you know, distinguish what rows are pass validation or not.

Q&A

Okay. So this is probably as far as I wanted to get with pointblank for R. Okay, good. Cause we have so many questions. I was just going to say. Okay, good. Okay, this is a great job for Q&A. Okay, great.

This, first of all, we all agree in the discord chat that this is infinitely cool and so amazing. And like the HTML widgets and actions and all of that stuff is just so cool. And I think it's a great way to kind of and so amazing. And like the HTML widgets and outputs are such amazing little quality of life things compared with like the raw scrolling that a lot of us are doing or the raw checks or individual assertions and stuff like that. So this is amazing.

We're all like all the applause, but I would love to get to some questions so we can help people understand. The first one is Lizette had sort of reworded her question about parquet so we could get a little bit more clarity around that. If you don't have clarity around it, we can follow up and that's okay too.

She was saying, do we need to read the parquet data into the R environment first? I understand from Rich's response that we don't like for databases. Oh, okay. Yeah, yeah, yeah. Okay, let me see. Just based on my own understanding with parquet, I think you do have to, cause it's just a file format. You have to read into a data frame first.

So that's like step before you can get to pointblank. So, and so once you read that into a table and I imagine DuckDB would be the same thing. Arrow table would be sort of same thing. Whatever can read parquet. It's almost the same as like reading a CSV. We're not looking at CSV directly. We're not looking at parquet directly. We're always reading into a data frame or some sort of table.

And does the difference come when you are using a database connection where you are like connected to a SQL database where things are actually happening on the database and not inside of R? Yeah, that's pretty much it. Like, yeah. The difference there is independent of how you get the data there, you're always running all the queries on the remote database. You're not pulling any large amount of data through the wire, like to your system.

So the checks are basically just like, they just get transformed into SQL check, SQL checks that run there. And they basically, it is a summary of like passing and failing. And then that gets pulled over. If there's any sort of pull and there has to be something, it's like two or three rows of summarized data. Like summarized, yes. Okay, great.

And as Hubert says, like it's great to have stuff happening outside. Yeah, yeah. Database whenever possible.

We had a question about Python that was in this same vein. That was, can we use the pointblank to validate data streaming over S3 or GCS bucket using pollers and the Python ecosystem? I did link to the pointblank in Python version. But is there any extra you want to add there?

Well, basically, let me show you that just so people know. There's a pointblank for Python. And if you just look through like the user guide, you'll find in demos, you'll find it works with all sorts of tables, pollers, pandas, anything that narbles supports. If you know that it's basically interchange, data interchange for data frames in Python. And it works with, of course, stuff that works on databases as well.

But we're talking about pollers here. And yeah, basically anything that works with pollers works here, including lazy frames as well. And although like if you do try it and somehow it doesn't work, let me know because it's not formally tested, but pretty much everything should work because it's being very simple with the pollers. Great, yes, that's great. If you do use anything, if something doesn't work, send feedback. That would be super helpful.

Integrating pointblank with Shiny

One was from Hubert about Shiny. Is there a way to integrate this into a Shiny app or could this be used to interrogate a Shiny app? Do you have any experience with people using it with Shiny or have you? Yeah, not myself, but I know Eric Nance used it in a table contest entry, like basically pointblank in Shiny. And there's a full word example and there's a repo there. He'd use it to validate podcasts, like every single podcast, basically like the podcast index. And so I know it could be done. And I've seen other instances where it actually is used in Shiny.

I think another pharma company used it and created a front end for this. And I believe there's been more than one of these as well. There's a few instances of that. Because basically you just have code here and you have objects and you can use them pretty much in Shiny as well because they're just simple outputs.

And if you don't know Eric Nance, he does the R Weekly podcast and maintains a whole bunch of stuff that's really beneficial for the R ecosystem. So everybody round of applause for Eric Nance. I'm hoping we can have Eric on actually at some point to talk us through Nix slash Rix, which would be great, but he's a busy dude.

Okay, let's move on. In the next couple of minutes, we've got a question from Dave. Do you know where those icons came from? Like we have the omega with the line through it, the little range and the little Lambda icon. He was asking like, are they Unicode? Like, what are they? No, I made them myself in Sketch. He made them himself. Yeah, I've got this little program here called Sketch. And basically I just draw and it's like really good. I mean, it's kind of like an old copy of it.

But basically, yeah, I just create like icons and I have like little, I just export them as PNG and SVG and then they just work. And I just draw a bunch of these, have a little template for these and they're sized really nicely and nicely exportable. And I also use them in Python as well for the same report. I basically replicate this report in Python as well and just move those assets over.

We have another one that is from Nathan that says, does the sundered data have a way to show which columns last row failed with which validation step? No, not that one in particular. Basically it's very summarizing and kind of filter. So you really don't know which like cell cause that row to be labeled as like bad or failed. Okay. Yeah, it's basically just like a quick utility for basically splitting.

You have to drill down. His question was like, for this data with the five columns, something that he is thinking of would output like a true or false column next to each of them. Sounds like something you could build, Nathan. Yeah, you could do that. Basically each step would, you'd have a vector of like true and false. You can get that from the object itself, from the agent object. So it's totally possible to make your original data set with that basically just like from that vector, make a new column.

I know everyone's probably like thinking about use cases for their own data. And to that point, Brent had asked because Brent works in the medical space. He says, I can see a great use case for this for medical device data with the internet of things or IOT. Has it been used for this purpose or have you seen anybody use it for that? There's actually a presentation at Posit Conf, I believe 2022, where someone presented on exactly, wasn't sure if it was medical devices, but it was definitely a medical setup. And it was an entire like half hour long talk.

So look for, look there. 2022, I believe that was like Chicago. And- It was DC. Chicago was 2023. Okay. Thank you. Yeah. Look up 2023 and look up pointblank. I was gonna say, I bet Isabel has got it already. There we go. Okay. It was Michael Garcia and it's scale your data validation workflow with pointblank and also there is praise for the animated hex, which is beautiful. It's gorgeous. If anybody hasn't seen the GIF of the pointblank, it's beautiful. Kind of scans through and shows all these little- Hard to get that on a sticker, I'm telling you. It is. It's like a lenticular sticker or something. I've been thinking about lenticular hex stickers lately.

Okay. One more. If we have just a quick second, Notobeko asked, does pointblank have a debrief function that gives you a summary of all the results from all the validation exercises?

Not really, but we have something close. What you can do is we have something called, okay, this is an embarrassing name, briefs. And basically it's something called an auto brief. This is very embarrassing. But what you can do is, I think if you just said, I forget how you do this. It's like brief true. There's a way to do this. And I feel a little embarrassed that I do not know it now, but you can have it. So like it automatically creates text here. It adds text. Basically like these lines right here will be inserted. We don't have a thing where like it just gets that into like, you know, like a data frame or, but I'm sure that can be rigged up because all that information is actually in the object. It's in there. It's in there somewhere. I think you could pull it out and make yourself a little table object, write yourself a custom function or add a PR package. Very welcome, yes.

Okay, amazing. Well, we've reached the top of the hour. People are hopping off to their next meeting, which I totally understand. Rich, this was so amazing. Thank you so much for joining us. Oh, thanks for having me. It was great. Oh, wonderful. Thank you everybody for joining us for asking amazing questions. I learned a ton more than I would have today because you asked great questions.

If you would like more Rich, go to positive confident person and say hi to him, go see him in Houston. But also he has given workshops on pointblank for different conferences throughout the years. First time I saw him, I think was at R Medicine or R Pharma. So there are tons of resources out there. The documentation for both the Python and R packages is beautiful. Go check out blastula, go check out gt if you haven't and Great Tables in Python. All right, everybody, we will see you next week.