Resources
May 6, 2025 |
1h 10min
|
15.2k views
|

885: Python Polars: The Definitive Guide — with Jeroen Janssens and Thijs Nieuwdorp

#Python #Polars #Pandas

Jeroen Janssens and Thijs Nieuwdorp are data frame library Polars’ greatest advocates in this episode with @JonKrohnLearns where they discuss their book, Python Polars: The Definitive Guide, best practice for using Polars, why Pandas users are switching to Polars for data frame operations in Python, and how the library reduces memory usage and compute time up to 10x more than Pandas. Listen to the episode to be a part of an O’Reilly giveaway!

This episode is brought to you by:
• Trainium2, the latest AI chip from AWS: https://aws.amazon.com/ai/machine-learning/trainium/
• Adverity, the conversational analytics platform: https://eu1.hubs.ly/H0jxK210
• Dell AI Factory with NVIDIA: https://www.dell.com/superdatascience

Interested in sponsoring a SuperDataScience Podcast episode? Email natalie@superdatascience.com for sponsorship information.

In this episode you will learn:
• (00:00:00) Introduction
• (00:04:46) Why Jeroen and Thijs wrote Python Polars: The Definitive Guide
• (00:18:18) Best practices in Polars
• (00:25:08) Why Polars has so many users
• (00:32:37) The benefits of the Great Tables package
• (00:50:05) Jeroen and Thijs’ partnership with NVIDIA and Dell for Python Polars: The Definitive Guide

Additional materials: https://www.superdatascience.com/885

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

It's if you think of an expression as a recipe, then the operations would be the steps, and the functions and methods would be the cooks. So how does this metaphor shape your philosophy about best practices with data transformation design in order to deliver clean, readable pipelines, especially in large collaborative projects? The short answer is no more brackets. With Polars, you take a different approach. You're writing a paragraph of things that you want to do. Yeah, and I think one of the things I've always liked most about Polars is the very declarative approach of what you're writing down. With Polars, you declare what you want as end result, and you just leave the specific processing and optimization to the engine, and that makes it way easier to read.

With Polars, you declare what you want as end result, and you just leave the specific processing and optimization to the engine, and that makes it way easier to read.

Jeroen and Thijs, welcome to the Super Data Science Podcast. You guys are together. I don't think... Have I ever had this situation before? I can't think off the top of my head of ever having two guests, but that they are co-located. Where are you two co-located? Well, thanks, John. It's great to be here again. Good to see you again. And we're calling in from Rotterdam, the Netherlands.

Nice. And that voice for the listeners out there, for the people not watching the YouTube version, that was Jeroen. That's his voice. And for people watching the YouTube version, he's the one in the pink shirt. Oh, also his mouth was just moving, and sound was coming out of his face. But whatever is easier for you to track. And then, in a charming and matching or complimentary forest green shirt, we have Thijs Nudorp. Thijs, what does your voice sound like? Thanks so much for having me, John. This is what my voice sounds like.

Oh, nice. I'm... You both have... It would be helpful if one of you didn't have a Dutch accent, but fine, we'll have to just work with this. We have accent? What do you mean, we have accent? Oh, God, no. That's funny.

So, Jeroen, you've been on the show before. You were on an episode 531 back in December 2021. That was the very end of my first year hosting this show. We had a great time on that podcast. It's a great episode that people can listen to. But Thijs, am I correct in understanding this is your first podcast ever? It is. Yes. I've never podcasted before. It's just since the book is taken off, we're finally getting into that marketing and we're kicking off with the best. So I have no clue what to do after this. I hate to let you know, it's going terribly so far. This is a bad episode.

It's too bad. Yeah, well, I'll just start with my biggest fear and just break it down from there, right? Your biggest fear. Yeah, exactly. We'll start with your biggest fear and then we're going to move on to your biggest beer. If you could grab one of those, I think it might smooth things along. Nice, nice.

Background on the guests and the book

All right. So you guys, until recently, until February 2025, you were co-workers together at Zomia, which is the leading data and AI consulting company in the Netherlands and makers of open source data frame library Polars. And then Jeroen, you recently took a DevRel job at Posit. It seems like a lot of people are moving over to Posit. A lot of big names are there now, makers of RStudio and lots of other great open source tools.

But what I'd like to speak about most in this episode is your new book, brand new, released actually the same day that we were recording this, which is April 1st. So now that this episode is live in May, hopefully it'll actually be available again because right now in a lot of locations around the world, at least, if you try to buy Python Polars, the definitive guide by our guest today, Jeroen and Thijs, you wouldn't be able to get it on April 1st because it is sold out. But O'Reilly are very good. They do on-demand printing. And so they should be able to resolve that pretty quickly. It's not like with a lot of other publishers. It could be months potentially before like a huge, I don't know, like a lot of publishing companies do orders in like the 5,000s, but I'm pretty sure O'Reilly, they can do printing on demand, which is cool. So anyway, that should be resolved soon. Very popular book. Very excited to have you on the show.

We've had Polars on the podcast before. We've had Richie Fink, its creator. We've had another key contributor to the Polars project, Marco Garelli. But the Polars library has grown a lot since then. It's about to pass pandas in popularity, if we measure that in kind of number of GitHub stars, if that's a measure of popularity. And yeah, and now it has this great O'Reilly book, thanks to the two of you.

Book giveaway

So what spurred you guys on to write the book? What was the experience? Like, oh, and I've got to tell the listeners about this, that you're doing a book giveaway. So I think we'll give them until Sunday. What do you think? Up until Sunday? Sounds good to me. Yep. Sweet. So there's a URL. You can say what the URL is and what the free book giveaway is. We do free book giveaways on the show, lots, physical books, but there's something special about your book giveaway that we've never done before. So I'll let you guys fill the audience in.

Yeah. For your listeners, we wanted to give away hard copies that are signed by the both of us. So in order to be eligible for a copy, you go to pollersguide.com slash SDS and you fill in your name and email address, and then you enter the raffle. And then by Sunday, we'll, yeah, we'll let you know. Even if you don't win, you'll still get the first chapter for free. Awesome.

That's a cool giveaway. I'll encourage more guests to do that. And so both of you will sign it. I guess if you're co-located, it makes it easier. Did I say three copies? I'm not sure if I said the number. I don't think you did, but yeah, three copies. We'll give away three copies. Nice. And people can be anywhere in the world? Anywhere. We'll take care of the shipping. Yeah. Sweet. Yeah. Super generous. Thank you very much for doing that.

You have until Sunday to submit yourself into the raffle and get a signed copy of Python Polars, the Definitive Guide from both of its authors. Our guest today, Jeroen and Thijs, super, super cool. All right. So yeah. So now with that out of the way, tell us what caused you to write this book and what the process was like.

Origin story of the book

Yeah. It started with you, right? Exactly. So let's start with the origin story here. I joined Xomnia in January 2022. Is that right? Yeah. No? No. Oh man. It sounds like he's asking somebody, but that's Jeroen asking himself. It sounds like he's having a conversation maybe with Thijs and me, but it doesn't really matter when. It doesn't really matter when.

So when I started, I was just getting to know everybody working in the office. And there was this one guy really focused working behind his laptop. Everybody was going to lunch, but he would just stay working. Turned out that was Richie Vink, the creator of Polars, I learned later. And I didn't know anything about Polars, but I didn't have an assignment yet. I had some time to work to explore a dataset, and I decided, let's try out Polars. And I was immediately hooked, of course. I immediately figured, okay, this is so cool. This deserves a book. This is going to be a big thing.

But I already knew, having written Data Science at the Command Line before, twice, that I never wanted to write a book by myself anymore. So I needed— Is that the tragic story of how you wrote it once and then accidentally burned it? And the dog ate his homework. I wish. No.

So I needed another victim, right? Someone to share the pain with. And Thijs, so very shortly after that, I got assigned at a client with a large code base, and Thijs was also working in that same team. So we were not only colleagues, we were also working for the same client in the same team. And so I felt like, hey, Thijs, Thijs, he seems to be good at this. He likes to write. Why don't I ask him? And to which his answer was, obviously, yes, yeah.

And so I had a meeting with O'Reilly anyway about whether I could do anything else for them, maybe a video course or something related to Data Science Command Line. But that's when I asked him, like, hey, have you heard of this thing called Polars? He said, yeah, yeah, we've had four proposals so far, but we've all rejected them. I was like, oh, wow, four proposals already. And so that's when I knew that we had to write a serious proposal. So we wrote one, over 15 pages. We brought in all the stats that we could, and of course, by then, O'Reilly hesitantly said yes. But after a few months, they realized, like, oh, wait a minute. This is actually going to be a big thing. We want to have this book now. So they started feeling the pressure, started asking, like, okay, about that deadline, like, is everything going all right?

Yeah. That's really scary, because writing a book is, it is torture. When I wrote Deep Learning Illustrated, it was the worst experience of my life. The only thing that came close was writing a PhD dissertation. But with a PhD dissertation, there's not that much pressure, because like two people are going to read it. You're going to have your PhD exam, like the board. You're the only people that are going to read it. And then also, there's this amusing thing of a number of girlfriends that I've had since doing the PhD, they, you know, this like early stage of dating, when they're like really excited about, you know, having met me, that goes away quickly. But there's this very brief window. You have an abbreviated PhD? No, but they like see it on the shelf, and they're like, I'm going to read that. And I'm like, you're not, you're really not. It's really not readable.

It's like, it's designed for a really specific niche of individual in the world that, you know, you have to spend like many years for this to make any sense. But then, but writing a book like Deep Learning Illustrated, the idea was hopefully more than two people would read it. And I don't know the whole time. I don't know if you feel differently about this, Jeroen, having now written several books before. So maybe you feel like, you know what? I can actually, I can write a bestseller. I know the process. I know what to do.

But at least for me, I mean, I've only, I've only released that one book so far. And for me, the whole time I was writing it, I was filled with this deep concern that it would come out and everyone would realize that I was a fraud, that like, I had no idea what I was talking about. I don't know if you, if you've had anything like that. Yeah, I recognize that, especially with the first edition of Data Science at the Command Line, which I wrote right after I finished my PhD thesis, I was in this groove. But I really felt like an imposter during that entire time, especially since everybody and their dog seems to have an opinion about Linux and Unix and which tools to use. And so a lot of opinionated people there, which made it all the worse.

But by the second edition, I kind of, I realized like, hey, you said bestseller. Well, I'm not sure that our book is going to be a bestseller. I'm pretty sure it's not, but. The Polars book? I mean, it might not be a New York Times bestseller, but I bet in some Amazon categories it will be. Maybe. Yeah. Yeah. It's funny how Amazon assigns these categories. Number one in database design. Yeah, you could be number one in graph database, exactly, graph database execution for children. Exactly.

But what I have learned is that I and Thijs, I knew that Thijs, we can definitely write a book, right? You don't have to know everything. That's what a lot of people think is that you have to be an expert in the topic. That's not true. You think, maybe you think that you're an expert, but as you start writing, you'll realize that you have a lot of gaps in your knowledge. And that's when you start learning more and more about the topic. So I was, by then, when we started writing Python Polars, I was pretty confident that as long as we would stay one step ahead, right? And what definitely helped is that we were able to implement the things that we learned at our client. And we can talk more about this later, how we actually put this into production. But you'll figure things out along the way.

You put the book into production? Not the book in production, but we have, I think that our client is probably that it's one of the first companies that has actually Polars code running in production. So it has that now for over a year before the 1.0 release of Polars. So yeah, so quite confident. And I guess the biggest takeaway here is that you don't have to know everything when you start writing a book. You'll figure things out along the way. Yeah, it turns out that the imposter syndrome is like, it's a natural part of the writing process.

Co-authoring dynamic

Nice. And Thijs, what is it like working with a tyrant like Jeroen? You want me to leave the room? Blink twice? Exactly. In my opinion, it went very naturally. I think quite early on, we already noticed that we have a relatively complimentary writing style that I just start putting words on paper and start restructuring and moving it around and refine it more. Something that stems from the time I was still writing a thesis where I couldn't get anything on paper because I was so judging everything you put down, like, nah, that's not quite it. And you kind of get stuck in that, right? So I learned to just get stuff out on paper and it may not be proper in the right format and the right semantics, not exactly the nuance you want to catch. But ultimately it gets you to where you want to be. It's just like the first 80% needs to come first and that's not perfect yet. And Jeroen is one of his qualities is that he can use his perfectionism in such a way that he's very good at the refining phase. So when I put some meat in the chapters already, he comes and moves stuff around and, oh, have you thought about this or shouldn't you word it like this? And that really does the eyes. So in that sense, not necessarily a tyrant, it's just a very effective perfectionist. Thank you. It's a fine line.

Yeah, there is a fine line and I am very well aware that there is such a thing as preparing too much, as overthinking things. And it really helps when there is already something on the page. So for example, that could be text written by Thijs or by myself. What I sometimes do as a trick, whenever I feel I'm stuck, because this is a book that involves a lot of code, I'll first write all the code cells, all the code chunks, so that I can then fill in the gaps with text along the way. That's one of a couple of tricks that I could apply here.

Very nice. How long do you think it'll be before book writing is just, is, is, will just be completely, you know, my next question is kind of a joke, but it's also, it's just such an annoying question and I'm, I'm even regretting that it's going to come out of my mouth. I'm going to do it because now I've like started down this road, but I was going to have that like classic thing. It's like when I'm having, when I'm out for drinks with friends that are, say, not data scientists or AI engineers, software developers, but maybe they listen to the podcast.

And so for example, I have a friend who's like, oh, you're really lucky that you got your book, Deep Learning Illustrated, out before the Chad GPT era so that people know you really wrote it. And then, so I was going to have this dumb, this question, which is kind of trite and you don't have to spend much time answering this. We can just get into some, we'll get into some Polars topics next, but you know, do you think there will be a time in the foreseeable future where O'Reilly just asks a machine for a proposal? They, it creates a 15 page proposal and then it says, you know, this is it. And then it goes and writes.

If you want to keep on regurgitating existing knowledge, then yeah, then using a stochastic parrot is great. But if you want to produce actually actual new knowledge, then I believe that humans are very much indispensable here.

But if you want to produce actually actual new knowledge, then I believe that humans are very much indispensable here.

Really a much, much better answer than I was anticipating, especially given the quality of the guests. April Fool's Day. Yeah, we are recording on April Fool's Day. And I guess it's still the morning in like Hawaii or something at the time that we're recording.

Why Polars is gaining traction

Let's talk about Polars. I do think your book is going to be a bestseller because there is a Polars moment right now, as I talked about kind of at the outset of this episode, with the popularity, at least in terms of GitHub stars, probably going to surpass the number of stars that Pandas has this year. Ritchie Vink and Marco Gorelli's episodes of this podcast last year were very popular in terms of both listens as well as social media reactions. And by the way, if people are interested, because we've been careful with this episode with the topics that we've curated, we won't be overlapping with Polars topics from Ritchie or Marco's episodes.

So if you want even more Polars after this episode, you can check out 827 with Ritchie or 815 with Marco or both. And I mean, they're outstanding episodes. Both are highly technical people, just like Jeroen and Tysar. And so we get with, you know, these are all complimentary episodes covering different aspects of the library. But yeah, very popular episodes, very popular social media reaction. I would not be surprised at all if your book did, if it sold like hotcakes. Everyone loves hotcakes.

The grammar of Polars: expressions as recipes

So let's talk about the grammar. So like R's tidyverse, which they make at Posit, where you now work your own, with Polars, there's also a grammar and a naming convention that is encouraged to preserve semantic clarity, which means that not only can you understand your code better when you come back to it later, but other people that you're working with can understand it more easily as well. In the book, you two compare expressions in Polars to recipes. So specifically, I'm going to read a little snippet of your book here. It's, if you think of an expression as a recipe, then the operations would be the steps and the functions and methods would be the cooks. So how does this metaphor shape your philosophy about best practices with data transformation design in order to deliver clean, readable pipelines, especially in large collaborative projects?

The short answer is no more brackets. When you read Pandas code, there are many brackets in there, and in a lot of cases, it's very difficult to reason about what the code is actually doing. And so with Polars, you take a different approach, not only with those expressions, which are indeed the building blocks, those small recipes, but also the part where you use the expression, namely in the entire query. So it's more a, it's almost like you're writing a paragraph, right? To come back to book writing, you're writing a paragraph of things that you want to do. It's a logical element in your entire pipeline. And that's much easier to reason about.

And I think one of the things I've always liked most about Polars is the very declarative approach of what you're writing down. So in Pandas, it can be the case that you're very focused on specific operations in parts of your data frame, which can make it hard to follow what exactly is going on in the hood. But with Polars, you declare what you want as end result, and you just leave the specific processing and optimization to the engine. And that makes it, it makes it way easier to read.

Maybe this is a good moment also to clarify that we are very appreciative of Pandas, right? We're not here to bash Pandas at all, right? That's what you sometimes see online, is these comparisons that are not done in a very, yeah. Not in an elegant manner. That's not us. Without Pandas, there wouldn't be Polars, right? So we are very much appreciative of what Wes McKinney and his team have done.

Standing on the shoulders of giants, right? Wes is now also, so Wes, the creator of the Pandas library, which for people who aren't already practicing data scientists, you may not be aware that Pandas has been for some time now, for a decade at least, the de facto standard for working with data frames, which are a kind of data that's like how you could put, you could imagine data are in a spreadsheet. So in like an Excel kind of tool where you can have columns that represent different kinds of data so you're not restricted to having just a matrix of numbers of flow values. For example, with a data frame, you could have similar to this idea of column names and you could have one column that's string information, like company names, another one that's number information, like how much revenue those companies had in a year.

And so for a decade or more, Wes McKinney's Pandas library, 15 years, Wes McKinney's Pandas library had been the standard for working with data frames in Python. And they are hugely important because you're constantly, as a data scientist or data analyst, you're constantly working with different types of data like that. And so working with them in Pandas has been key. But Polars has taken off, again, like hotcakes. It's burst onto the scene recently. Ritchie has led development of this, Ritchie Vink.

So I realize that you guys don't, it's not nice to bash Pandas, but why are so many people switching over to Polars today? Like what's the kind of nuanced argument that even maybe Wes McKinney himself would forward? So I think one of the, when we were talking with Ritchie over the many times we talked with him over the course of writing the book, one of the main experiences that shaped how he wanted Polars to work was some frustrations that he had when running his pipeline and only 20 minutes in, you run into some trouble and it crashes. And that's not something you could have seen up front.

So this is one of the things that, one of the experiences that helped him shape what Polars ultimately became. And there's also a lot of good things that he saw in how Pandas works that he wanted to take and put in Polars. But generally, like Pandas became a big inspiration, both good and bad for Polars and also other libraries. I think like Spark, especially you can see that the syntax of the Spark is a lot like how Polars turned out. And there's other elements, like for example, from the Rust language that he, that Ritchie took to implement in Polars because it just made it work so nicely. He talks a lot about Rust in his episode.

Polars in production: the Alliander project

So there's a lot of foundation around your book and around Polars and why people are using it so much for DataFrame operations today and more and more and more so. Earlier in the episode, you mentioned about a real world implementation of Polars and maybe, as you said, maybe the first ever production instance of Polars. And so am I right in understanding that's Alliander? I'm probably butchering the pronunciation of that. Alliander. It's a power grid provider in the Netherlands. So they provide the infrastructure for both electricity and gas in a third to a half of the Netherlands, I believe.

So the largest utility company in the Netherlands, therefore, I can't even say Netherlands. It's Dutch pronunciation. Nederlands. So tell us about that project. And actually, it'd be interesting to know, like, when was there overlap in working on the book, in working on that project, and did working on a Polars book help with a real world implementation?

The origin story here is that Thijs and I, we were both very excited about Polars. We were writing a book about it. And then, all of a sudden, it became clear that at Alliander, we needed to speed up the pipeline. We needed to lower costs. We needed to process much more data. And in the current state, that just wasn't possible. It was a combination of not only Python and Pandas, but also R code. So it was very inefficient. To give you an idea, we were running this on a single AWS instance that had over 700 gigs of RAM. 700 gigs of RAM.

And so, yeah, we can provide you a link with more backstory to this, with some actual numbers. But we were very excited, and we were like, hey, let's try this out. Let's do this. At first, the team was very hesitant. We had two people, or three actually, we had another colleague, three people promoting Polars that is being developed at Xomnia, right? So they were very skeptic, understandably. So what we did in order to convince them is to just take on a very small piece of code, some low-hanging fruit, and benchmark it, and re-implement the Pandas code into Polars, and then just show the numbers. And by then, they were immediately convinced, right, this is indeed way faster, uses way less memory. Let's try this out.

Let's take on this huge code base, piece by piece, by translating, not one-to-one, because you can't do that. You really have to reason about the inputs and the outputs, and then do it in an idiomatic way, right? You cannot just translate Pandas to Polars. I think it took us, well, what, six months, a year? I don't even remember. But eventually, I left that client at that time, but there was a moment like, okay, we can now get rid of R and Pandas as a dependency of this project, and it's been running smooth ever since.

Yeah, I think ultimately the size of jobs at the beginning was about 500 gigabytes for just that task of doing like one calculation, and we shrunk it down, like both being a consequence of implementing Polars, but also as we were going, rehashing some of the code structure that we were using in the project, we hashed it all the way down from 500 to 40 gigabytes, which makes it a lot more doable to run calculations.

Yeah, I think ultimately the size of jobs at the beginning was about 500 gigabytes for just that task of doing like one calculation, and we shrunk it down, like both being a consequence of implementing Polars, but also as we were going, rehashing some of the code structure that we were using in the project, we hashed it all the way down from 500 to 40 gigabytes, which makes it a lot more doable to run calculations.

So the second part of your question was like, okay, how did this influence each other, the book writing and putting it into production? And this was, yeah, it was a perfect match, because when you're actually need to put it into production, when you have a real problem to solve, that's also when you start to notice the limits, right? Or maybe inconsistencies or missing functionality. For example, there was this random sampling with weights, right? That's something that you can do in Pandas. You just give it another column that indicates the weights for the sampling. That's something, maybe even up until this point, something that Polars doesn't have. Luckily, that was for an ad hoc analysis that we had to do. But at that point, it becomes clear what Polars can and cannot do.

Also, when you write, you start to look at things from a little bit of a higher level. So sometimes we noticed inconsistencies in naming or missing methods, like, hey, why is there no inline operator for the XOR operation, right? That's something that nobody ever thinks about, but when you need to put in a table in your book and you need to fill in all the pieces, that's when you start noticing these kind of things. So we were able to also submit some issues, maybe even a few pull requests to Polars itself along the way.

Very cool. So you're actually influencing the library itself as you're writing a Polars book, as you're working on consulting projects, bringing Polars into the real world, getting huge benefits in terms of memory footprint, that 10x figure that you gave there, 500 gigs of memory down to 40, that's massive, definitely makes it a lot easier to be working with the data.

Do you happen to have, and no pressure if you don't, but it was nice to get that 10x for memory, that 10x improvement. Do you happen to know what it was for compute time? Was it about 10x as well, kind of thing? I think ultimately the compute time, because along the way, one of the things why we had to optimize the code was because the requirements for the amount of samples that we were running for a certain simulation were supposed to hit 50 samples, it was like what the stakeholders asked us to strive for. And the 500 gigabyte instances was already 25 samples, so we couldn't push it higher because it just stacked higher and higher. And that's, at the end, ultimately we were able to do those 50 samples in the same time frame that it took to do the 25 samples at the beginning.

Styling tables with great-tables

I want to move on to a slightly different topic from your book, but it's related to this idea that you just mentioned around improving code bases, improving the Polars library, and kind of the flexibility, the full breadth of capabilities that it has. In your book, you introduce a way to style tables with a package called great underscore tables. Great tables. But yeah, if you're typing it out, the package is great underscore tables. And in a talk in your room, you actually mentioned that tables are underappreciated in visualization. So could you elaborate on why this great-tables package was created? What it does? Yeah, maybe what its advantage is to existing approaches out there?

In hindsight, so now that this package exists, right? Great tables. It's strange that there wasn't already a package, right? Because tables are everywhere, especially when people are working with Excel, a lot of people really like to add styling to this in order to make it presentable to stakeholders. Add some color, use currencies, what have you. Maybe some mini graphs in there, right? So now that it's there, it's so obvious that there should be a package for this.

So Rich, I'm actually not sure how to pronounce his last name. The creators of great-tables, Rich Eon, I'm butchering that. But I do know the co-creator, well, they're both my colleagues, I should know, but I just call them Rich and Michael Chow. Great folks, you should have them on the show as well. They created the great-tables package.

And so just only a few days ago, I saw a post by someone about Polars advocating or actually recommending that, okay, it's useful to add in the dollar sign when you're presenting currency. But what he was doing, he was actually changing the underlying data. I'm like, wait a minute, that's not the way to do it. You want to change how it's represented, right? You want to put this layer on top of it. That's what you need to do. And that's what great-tables can provide. So you're not changing those floats or integers to strings in order to format it. That's not the way to do it. So there should be another layer. And Python has a myriad of data visualization packages. But when it comes to producing tables, well, I only know of one, and that's great-tables.

And Python has a myriad of data visualization packages. But when it comes to producing tables, well, I only know of one, and that's great-tables.

So with Polars, you can indeed style data frames using the great-tables package created by Rich Yannone and Michael Chow. So you use the DF style accessor, and that will then use the great-tables package under the hood.

There you go. And so to maybe, I'll try to explain back to you the example you just gave me there with the dollar signs. And you can tell me if I'm getting this right, that basically you're saying, if you have this huge, it doesn't matter if you have a very small table. If you think about a spreadsheet with 100 rows, it doesn't really matter if you write some kind of find replace that goes and adds in dollar signs at the beginning of every number in a column. But if you have a gigantic piece of data, then trying to edit each of the individual items in that gigantic column would be very computationally and memory expensive. And so with great-tables, you have this abstraction above where you don't need to individually change that information in all the rows. It's just, it's like an attribute of the column that changes.

Yeah, I wasn't really hinting at the performance issues right there. Just it doesn't feel right to me that you're changing the actual data. You want to keep the data, the data, because you never, you never know what you want to do after that. Maybe you want to have a subsequent calculation going on. You have to strip the dollar sign again. Maybe you want to have round numbers or yeah. So there's so many, many instances where you just want to change the representation and keep the underlying data intact.

The Python Polars Definitive Guide and UV package manager

So your book, listeners should definitely check that out. The Python Polars Definitive Guide, I should say it properly, Python Polars, The Definitive Guide. The Definitive Guide. And yeah, that's out on O'Reilly now. You can get it anywhere in the world. And if you're listening to this the week that this episode has come out, you can also head to pollersguide.com slash SDS to go to the raffle to get a signed copy by both your own and Tice, which is awesome.

I'd like to move now on a little bit to topics beyond your book. So we have been talking in this episode obviously about pollers a lot, which is a popular Python package. But another Python package that is really taking off recently is UV. And so it's a Python package and project manager that climbed from zero GitHub stars to, yeah, to, I actually don't have the number in front of me right now, but a very large number. Lots of people are talking about UV. And it has exceeded poetry as another longtime favorite for package management. So Tice, in a blog post, you mentioned ditching poetry for UV. You talk about increased speed, reliability, and ease of use as the reasons for that. Do you want to tell us more about UV, poetry, and whether people should be calling it of?

Gladly. So this is also one of the things that we decided to do for the book. We started out with poetry and did all the version management of Python versions with buy-in and other tools around it. But ultimately when we were prepping the repo that can be used by the readers of the book that contains all the different notebooks that come with the chapters, so you can follow the chapters along and execute the code yourself, play around with it, obviously you need to set up an environment easily that can work on many different systems that all your readers might have. So in the beginning we were thinking maybe to go for Docker because that generally is the easiest way to make something run on different kinds of configs. But as we were writing the book, UV became bigger. And at one point I just started experimenting a little bit with UV to see how easy it is to set up. And it boiled down to installing UV and then running UV sync, and it sets up everything. It sets up the right Python version, it just finds the right dependencies for your system. Everything just clicks. So that's ultimately what we went for as the final solution for that repo, to allow people to just install UV and just make it work.

And one of the reasons I started playing around with UV was mostly because it goes with the trend of the Rust-based tooling, which shows that very much like the performance of tooling is a feature in itself. It's one of the things that Polars showcases and it clicked very well, like UV has the same kind of thing going for it. Also the Rust-based tooling, which is leagues faster, it can be more than 10 times faster. That combined with the single command setup, it's made it a very quick and easy win.

Maybe you can say a few things about the regression that you found in Polars, where you came in handy. Yeah, so at some point, UV is so fast that you can on the fly set up an environment, like an ephemeral environment that's just set up for just that command and then torn down again. Yeah, so I was playing around with that to benchmark the different versions of Polars to see what the speed is on different queries, different kinds of setups, and iterating over the versions and just bumping it every time to see what happened. At one point I found, I think in version 1.2 point something, that there was suddenly a regression. The query started taking 10% longer to run the full benchmark, and it didn't really go down again. And drilling down, we were able to pinpoint the two specific queries of the benchmark that we were running just spiked up on a certain version. And because UV just sets it up so quickly, at one point with a script for git bisect, which allows you to pinpoint the exact commit version in the Polars repo where it started occurring, allowed us to find which specific commit caused this regression. And funny enough, when I communicated it to the Polars guys in that week, they hit the same code. For some reason, they couldn't quite figure out quickly what exactly caused it, but they hit the same code and refactored it and it resolved itself. So ultimately, all was good.

But it was interesting to finally have a package manager that was able to be used so quickly that you can start using it for complete new use cases that you couldn't have thought of before. Nicely said. I haven't been using UV myself yet, but it sounds like I should be. I can definitely recommend it.

The command line and R packages

This one, this next one is maybe, well, this is a bit targeted more at your room, though, Tyse. You might have lots of opinions about this as well. This is about another open-source question, another open-source library, well actually like a whole open-source language, which is the command line. So like bash at the command line. And so your whole previous episode on this show, 531, was all about data science at the command line. Obviously, you've written two editions of the book, as discussed. You've written R packages that make the command line more interactive and playful. I don't know if I can pronounce them properly. There's Rayliber, R-A-Y-L-I-B-E-R. Rayliber.

Well, Rayliber is a wrapper around Raylib, which has nothing to do with the command line, but it's a C library to create video games. To create video games. And I have actually, I have a talk, I've given a talk at NYR a couple of years ago, where I advocate for some of the things that video game programming offers, like 2D and 3D graphics and interactivity. How that can be used for doing data science. But that's, so that's Rayliber. That was a fun project, right, that you can actually create 3D environments from R.

But it has nothing to do with the command line. So let's talk about the command line. I mean, I, we, yeah, there's, I don't know if any of the other, the Rexpect or Tmuxer, if those have anything to do with the command line. Oh, yeah, yeah. So Tmuxer, that's a wrapper around Tmux, you know, the terminal multiplexer. If you want to run multiple terminal sessions at once, at like these sessions, and you can interact with that programmatically from R using the Tmuxer package.

Now, that's actually a triad of packages that you just, well, you mentioned two of them. So Tmuxer, there's Rexpect, which is from R-expect, I'm not sure if you're familiar with the expect. I'm not. No. That allows you to automate things on the command line. So log in to a server automatically and then do certain things based on certain outputs. Now, I wrote a wrapper for that. And then there's also knitter-active. And now I'm probably the only one who has used these packages at all, but I needed those. I mean, that's, of course, why you should write software in the first place for yourself. But I needed those three packages in order to be able to write a book about the command line using bookdown, using knitter, which is the system that I used at the time. So a lot of, what's that called, yak shaving or bike shedding. Lots of work, not actual writing, but lots of work. And we had some of that for the Polar's book as well.

But yeah, I mean, when you're an engineer, when you're a developer and you're writing a book about development, there's always some kind of developing that you need to do on the side for the book, whether that's just to get in the groove or whether it's actually helpful just to make life easier. Yeah. Yeah.

Thank you for that extra context and explaining what those packages do. And all of that was basically just to bolster in a few minutes, your expertise on doing data science at the command line as a real expert on that.

So something that I want to highlight here is that in your course, Embrace the Command Line, which folks can check out, it's online, and there's more information at yaronjansens.com slash embrace. Well, I'm sorry to interrupt you. No, please do. This was a course that I've given a couple of times. This was a cohort-based course using Maven, and I've given it a couple of times and I no longer do it. So unfortunately, it's not available online.

We can cut that out. No, it's okay. You can just leave it in. I don't mind my mistakes being on air. Neither do I. Nice.

But nevertheless, in that course, or at least in the course information, you say that the command line is as powerful as it is intimidating. So for our listeners out there who maybe haven't crossed that emotional barrier, maybe they do program, they use Python, maybe they use R, or whatever programming languages they use, but they haven't crossed that threshold, that emotional barrier, to start using the command line, what do you recommend to students to kind of get past that emotional barrier and see the command line shell as a great creative space for data science and software development?

Embracing the command line

Yeah, it's unfortunate that when you first see this window, this terminal, this blinking cursor with a prompt waiting for your commands, that it's such a shame that this is indeed so intimidating. But that's how, of course, when Unix or Linux was first created in the 60s and 70s, at that time they didn't even have screens, so things had to be, they weren't all flashy. So there is indeed a hurdle for you to take, for you to embrace the command line.

And there are certain tricks that you can apply, certain changes that you can make in order to make the command line, yeah, a more pleasant environment, a more forgiving environment. So things that I always like to do are, let me try to come up with a couple of them. First of all, use colors that you like, use a font that you like, add in aliases so that these long commands, these long incantations, that you don't have to remember them by heart, so you make the experience more ergonomic.

It also helps to work in an isolated environment so that you know that you won't be able to break anything, Docker can be used for this. And I think if you do these kind of things, experiment with the command line every day for a little bit, don't try to do everything all at once, I mean, I don't, I just use it here and there, right, as a complementary set of tools in addition to, well, all the other data science tools that you want to use. And then, yeah, you'll gradually build up more and more appreciation of the command line, you'll be able to embrace it more and more, make it your own.

It also helps to work in an isolated environment so that you know that you won't be able to break anything, Docker can be used for this. And I think if you do these kind of things, experiment with the command line every day for a little bit, don't try to do everything all at once, I mean, I don't, I just use it here and there, right, as a complementary set of tools in addition to, well, all the other data science tools that you want to use.

Dell and NVIDIA collaboration

And so I realize now I've strayed, I've already kind of switched gears and taken us away from your Fuller's book, but I remember now that there were a couple of stories that we discussed before coming on air that I really wanted to cover before this episode ended. So we're going to have this grating experience for the audience of going back to your book, but maybe that's a nice place to end anyway.

And so first what I wanted to talk about, and so you guys may or may not be aware of this, but in 2025, two of the biggest sponsors of this podcast, to whom we're very grateful because it allows us to keep the lights on and make this show for everyone, are Dell and NVIDIA. And it sounds like for the appendix of your book, Dell and NVIDIA, you had some kind of partnership with them that allowed you to do more, explain how they were involved with your book.

Yeah. So at a certain point, I got a LinkedIn message from NVIDIA. It was something about being an influencer, and at first I didn't think much of it. After a week or two, I decided to reply like, all right, I'm interested, let's chat. And it turned out that they actually wanted to collaborate with us. They were quite eager to send us some hardware so that we could benchmark colors on the GPU. And we're like, great. Only thing is, we don't have anything to put that video card in. So that's when they brought in their partner, Dell, and Dell was able to supply the rest of the hardware.

So that was a fantastic collaboration. And the way we did this is, Ty can say more about the software side of things, but in terms of hardware, it was all in the States. So Dell had this laboratory where they had, you know, a beefy machine, and they were able to swap out, you know, different NVIDIA video cards. So we did the RTX 6000, no? The Ada generation. The Ada generation. So these were all professional video cards, not the consumer. Yeah. So the workstation variants.

So, yeah, we were, it was very important for us that we were able to benchmark things ourselves, that we wouldn't just copy numbers from some, you know, some leaflet, right? Some promotional material. We wanted to produce these numbers ourselves if we were going to put them in our book. And you know, that was all fine. NVIDIA and Dell thought it was a great idea. And so eventually we were able to try out five different video cards for a number of different settings and packages. And that's all reported now in the appendix of the book.

But Ty, maybe you can say something about how you actually benchmark. Yeah. So to start off with a little more context is that NVIDIA has a team called Rapids, which is working on creating all kinds of general purpose computing packages that can run on the CUDA platform. And CUDA is like the platform, calculation platform that NVIDIA opens up. So you can run any kind of calculation effectively on the GPU. And the difference between normal CPU and GPU is that GPU has many relatively dumb, simple processors, but just many of them. So if you're able to bend a calculation problem into something that the GPU can run, it oftentimes accelerates by a lot, by a factor up to 10.

So they also did this for packages like Pandas. They have CUDEF, is what their package is called. It's a data frame library, but runs on the GPU. And they wanted to collaborate with Polars as well. But since Polars has this layered architecture where it runs through an optimizer first and only then gets sent to an engine, it would be a waste to just put the Polars API on CUDEF and just translate it to normal CUDEF functions, because a lot of the performance enhancements from Polars comes from this optimization. So instead, Rapids worked together with Polars and designed a GPU engine that gets input from that optimization layer.

Because they recently finished the open beta for this new package, they got in contact with us to ask, hey, you guys are working on the book of Polars, do you want to collaborate? And we said, well, with the terms that we could test stuff ourselves and benchmark ourselves, we definitely said yes, because it turned out to be a lovely collaboration as well.

That's a cool story. Yeah, definitely. What were the results? Is that what you're going to tell me now? Please tell me the results. It's a lot faster. It is. Yeah. So we already noticed that in the beginning, the promotional material was a bit careful with what kind of size of data set it would be beneficial from. And it turned out from the test that we were doing that it's quite quickly already. Because data needs to be transferred to the GPU, you get a small overhead. So you start seeing the difference when the data set size grows. But it's already from one gigabyte and up, so it's relatively quickly because most data that you would work with in a professional setting usually tends to grow a lot.

And we also noticed that even the relatively smaller GPU cards with less processors already benefit a lot from this, already have a big speed up from just using the GPU engine. Very nice. Cool project. Great results. Unsurprising results, given everything that we know about Polars already, including the examples that you gave at Aleander earlier in this episode. But cool to have that comprehensive benchmarking there on four different NVIDIA cards. And cool that Dell supplied the server for you to be doing all that benchmarking yourself.

Marco Gorelli and the chapter rewrite

And then one final story that I want to get in here. So I mentioned already how Marco Garelli, so Marco Garelli was our first ever Polars episode on this podcast. So that was episode 815. And then he introduced me to Richie, the creator of Polars, who came in not long after that, a couple months later in episode 827. And now you guys are our, you're the final episode in the trilogy on Polars, well probably not the final. We'll have more. But for this, it's like the original Star Wars, 4, 5, and 6. You're episode 6 of the original Star Wars.

So I understand that there's an amusing story involving Marco somehow sabotaging your book and forcing you to rewrite an entire chapter. Yeah, it's amusing now. So we have to go back a little bit further. Was at a Christmas party organized by Xomnia, where Richie was also present. And Richie was like, yeah, Polars is going to have data visualization capabilities. I'm like, what? Python doesn't need another package to do data visualization. There are already two dozen, so many out there.

So at first I was like, man, they keep expanding the library. We just want to finish this book. So I was quite upset at first. After a while, I started to realize, okay, maybe it's not so bad. I mean, if the book has a chapter about data visualization, maybe it'll sell better if it has pretty pictures. So I started writing. I was quite happy to find out that Polars itself doesn't do any data visualization. It has the df.plot namespace. But then every method in that namespace calls out to another package, hbplot. And I wasn't familiar with hbplot yet. It's this meta package which can target matplotlib and plotly and another one, bokeh. Thank you.

And so, okay, I really had to get into hbplot, but I didn't just want to write about hbplot. I also wanted to include great-tables, right, was a big fan of that. And you could argue that presenting a table is also a form of data visualization. I'm a big fan of plot 9. So it was going to be a huge chapter. So I've written this, and then all of a sudden I see on GitHub this pull request by Marco Gorelli. And he was like, okay, I'm going to change out hbplot for Altair. I'm like, what? Now I need to rewrite the entire chapter or at least a big portion of it. Like, Marco, what are you doing?

Now I know that Altair is a very good choice for this, especially for when you are working in a browser and you want to create interactive data visualizations, right? That is something that plot 9, for example, doesn't support. So Altair definitely has its use cases. And I should have known better as well, hbplot at the time, or the whole plotting functionality in Polars was marked unstable. So I should have known better. I was just too happy to get it out there. And you know what, Marco and I, we get along really well. We collaborate now on getting Narwhals, right, his project into plot 9 so that plot 9 better supports Polars as well as Pandas.

But yeah, that was the story that I had to rewrite nearly everything inside chapter 16 visualizing data. Nice. Great story. And it is funny to imagine Marco kind of sabotaging your book, because he's an extremely nice because we actually, the episode with Marco, I don't know if you know this or not, but I recorded it with him in person in London and he took a train, he took a train from Cardiff to London, which is like three hours or something to come and record the episode.

And and then so I, we went for dinner afterward as well. And you really get this impression of a man who is exceedingly kind and kind. Yes. Yeah. I hate it. I hate it. I wish he wasn't like that. No, he's a very generous and kind person. Definitely a pleasure to work with.

And I definitely love his dry sense of British humor. It's perfect. Every, every, every time he speaks at a conference, he tries to incorporate an expression from that country. So when he was presenting at Pi Data Amsterdam, he used the expression, helaas pindakaas, which translates to too bad peanut butter. Doesn't make, doesn't make any sense if you're not Dutch. But that's what I, and in Germany, he talks about the concept of luften or wuften and right to open up all your windows. And I think in Pi Data in Paris, he had a data set where about the pronunciation of pain au chocolat. And when it was at one point in the further south, you go, it changes something else. It's something like that worked in. He's a funny guy. Definitely. Yeah. That's really funny.

So people want some more of that humor. And he, he was also, he was extremely technical. I mean, the, the depth, you know, in this episode, we haven't gone and that wasn't really the point of the episode in that one as well as Richie's episode. So at 815 with Marco Garelli or 827 with Richie Vank in different ways, you get into the nitty gritty of why Polar's is so fast under the hood. And so if people want to check those out, we'll have links to those in the show notes.

Closing remarks

Um, all right. And so that brings us to the end of this episode, basically. It's been awesome having both of you on the show. Jeroen, welcome back. And, uh. It's been a pleasure. Yeah. Hopefully we'll have you back again soon. Thijs, I hope your first podcast experience wasn't too painful, uh, to, to kind of end things. Great.

Book recommendations

Um, Thijs, let us know if you have a book recommendation for us to, to wrap things up here. I do. Yeah. So, uh, besides normally I'm more into the fantasy side of things to escape, but, uh, one of the things I love is when someone is able to explain a complex, uh, something complex in a way that has a proper story. And one of the books I recently read was, uh, immune by, uh, the Philip Detmer, the, the main writer of Kurzgesagt, which is also a YouTube channel, which says, which has many explanation videos on all kinds of intense topics. But this book, uh, he dives into the immune system, which is exceedingly complex, yet he still is able to, to explain it very, very well in a very both informative and entertaining way. So that's definitely one of the books I recently finished that I love reading.

I love that recommendation. It has been advertised to me at the end of a number of Kurzgesagt videos. Kurzgesagt is one of the few YouTube channels that I subscribe to and it is excellent. I mean, if I could, if I could like go back in time and somehow be responsible for one YouTube channel, it would probably be Kurzgesagt. I think it's amazing. Good choice.

Um, and, uh, for our listeners out there who don't speak German, it's Kurzgesagt means shortly said. And so it is just like you described his immune book being, uh, you know, well-spoken, easy to understand on a complex topic. That's what all the videos on the channel aim to do. And I highly recommend, uh, Kurzgesagt. So many fascinating topics, scientific ones, but also philosophical ones. It really gets into the big questions of life in the universe. Interesting. Nice.

And Jeroen, do you have a book recommendation for us? Sure, I do. I am currently enjoying Unix, a history and a memoir by Brian Kernighan. It's a short book. I haven't finished it just yet, but you know, as we have talked about, Unix can be a very dry topic, but it is so interesting to learn about the history and the people behind it and the politics and the things that went on in, yeah, developing, uh, such a, uh, profound piece of, of, of software. Yeah. Truly revolutionary. It doesn't, it doesn't understate it.

it is so interesting to learn about the history and the people behind it and the politics and the things that went on in, yeah, developing, uh, such a, uh, profound piece of, of, of software.

If you're watching the video version, uh, you can see what the book looks like. Uh, but yeah, truly revolutionary, uh, Unix is, I mean, it's, it's staggering to think the shoulders of the giants that we now get to stand on, um, you know, inventing our relative trivialities, uh, in computing, which although interestingly, even though they're relatively trivial, relative trivialities compared to say Unix, because of what's to come in terms of this age of intelligence that we're emerging into with intelligent machines, they could, you know, maybe in decades they'll be looking back and thinking about great-tables and the big differences, you know, the history of great-tables book.

How to follow Jeroen and Thijs

Um, guys, it's been so great having you on the show. If folks want more of your humor and brilliant insights, how can they follow you after the program? Well, the easiest way is probably go to the same website that we already mentioned, Polarsky.com. From there on, you can find links to our LinkedIn pages and other places where we are. Yeah. That's probably easiest. Cool. Yeah.

And so basically you would say LinkedIn is the main social medium for you both? Yeah. I'm, I'm trying out blue sky. Um, I, I've been pretty active or I've been enjoying Twitter for a long time, but that has, uh, changed, uh, LinkedIn. We seem to get a lot of response, uh, on LinkedIn. Whenever we post something about the book, there's a, there's a, you know, there's a good vibe. I mean, there's a lot of other stuff going on on LinkedIn, but, um, yeah, it works. It works.

Yeah. LinkedIn is working these days, which we can't say for every social media platform out there. Uh, it's definitely where you can find me, uh, the most active as well. Awesome. Uh, thanks so much. This was a, another great episode, uh, really had an awesome time with you guys. Appreciate you co-locating yourselves in the, uh, well-appointed perfectionist tyrant, your own studio over there. And, uh, yeah, I look forward to welcoming you guys on the show again sometime. Thanks John. Thanks so much for having us.

Episode recap

What a tremendous episode with Jeroen and Thijs. In it, they covered Python Polars and how it's a high-performance data frame library that uses a declarative approach, allowing users to state what they want as an end result while the engine handles optimization. We talked about how the book Python Polars, the definitive guide by Jeroen and Thijs provides comprehensive coverage of Polars and includes benchmarks showing Polars can reduce memory usage and compute time by up to 10 acts compared to pandas, the standard, well, or at least until recently until Polars came along the standard dataframe operations library. Polars uses a grammar where expressions function like recipes, operations are steps and functions or methods are the cooks creating readable code without excessive brackets.

We talked about how when implemented at alley under a Dutch power grid provider, Polars reduced memory requirements from 500 gigabytes to 40 gigabytes. So a 10 X reduction and doubled processing capacity. We also talked about other things than Polars. So we talked, for example, about how the UV package manager is emerging as a faster rust based alternative to poetry, allowing for quick environment setup and tear down for benchmarking or whatever it doesn't need to be for benchmarking. It can be whatever reason you would need Python packages. And finally, we talked about great-tables and how it provides styling capabilities for data tables, allowing presentation ready formatting without modifying the underlying data.

As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Jeroen and Thijs social media profiles, as well as my own at superdatascience.com slash 885. And if you'd like to engage with me in person as opposed to just through social media, I'd love to meet you in real life next week at the Open Data Science Conference ODSC East running from May 13th to 15th in Boston, Boston. I'll be hosting the keynote sessions and along with my longtime friend and colleague, the extraordinary Ed Donner, I'll be delivering a four hour hands on training in Python to demonstrate how you can design, train and deploy cutting edge, multi-agent AI systems for real life applications. That is going to be fun.

Thanks, of course, to everyone on the Super Data Science podcast team, our podcast manager, Sonny Breivich, media editor Mario Pombo, Nathan Daly and Natalie Zajski on partnerships, our researcher Serge Macice, our writer Dr. Zahra Karshay and our founder Kirill Eremenko. Thanks to all of them for producing another laugh filled episode for us today, for enabling that super team to create this free podcast for you. We are, of course, deeply grateful to our sponsors. You, listener, you can support this show by checking out our sponsors links, which are in the show notes. And if you yourself are interested in sponsoring an episode, you can get the details on how to do that at johnkrone.com slash podcast. Otherwise, share, review, subscribe, edit videos into shorts to your heart's content. But most importantly, just keep on tuning in. I'm so grateful to have you listening and hope I can continue to make episodes you love for years and years to come till next time. Keep on rocking it out there and I'm looking forward to enjoying another round of the Super Data Science podcast with you very soon.