Transcript#
This transcript was generated automatically and may contain errors.
Hey, Wes, welcome to Talk Python to Me. Thanks for having me.
You know, honestly, I feel like it's been a long time coming, having you on the show. You've had such a big impact in the Python space, especially the data science side of that space, and it's high time to have you on the show, so welcome. Good to have you.
Yeah, it's great to be here. I've been heads down a lot the last N years, and I actually haven't been, because I think a lot of my work has been more like data infrastructure and working at even a lower level than Python, so I haven't been engaging as much directly with the Python community. But it's been great to kind of get back more involved and start catching up on all the things that people have been building. And yeah, so being at Posit gives me the ability to sort of have more exposure to what's going on and people that are using Python in the real world.
Yeah, there's a ton of stuff going on at Posit that's super interesting, and we'll talk about some of that. You know, sometimes it's just really fun to build, you know, and work with people building things, and I'm sure you're enjoying that aspect of it. For sure.
Wes McKinney's background
Nice. Well, before we dive into Pandas and all the things that you've been working on after that, you know, let's just hear a quick bit about yourself for folks who don't know you. Sure. So, yeah, my name is Wes McKinney. I grew up in Akron, Ohio, mostly, and I got involved, started getting involved in Python development around 2007, 2008, and I was working in quant finance at the time.
I started building a personal data analysis toolkit that turned into the Pandas project, and then open sourced that in 2009, started getting involved in the Python community, and I spent several years writing my book, Python for Data Analysis, and then working with the broader scientific Python data science community to help enable Python to become a mainstream programming language for doing data analysis and data science. And in the meantime, I've become an entrepreneur. I've started some companies and have been working to, you know, innovate and improve the computing infrastructure that powers data science tools and libraries like Pandas. So that's led to some other projects like Apache Arrow and Ibis and some other things.
And, yeah, and in recent years, I've worked on a startup, Voltron Data, which is still, you know, very much going strong and, you know, has a big team and is off to the races. And I've had a long relationship with Posit, formerly RStudio, and they were, you know, my home for doing Arrow development from 2018 to 2020. They helped me incubate the startup that became Voltron Data. And so I've gone back to work full time there as a software architect to help them with their Python strategy to make sort of their data science platform a delight to use for the Python user base.
About Posit
I'm pretty impressed with what they're doing. I didn't realize the connection between Voltron and Posit, but I have had Joe Chung on the show before to talk about Shiny for Python. And I've seen him demo a few really interesting things, how it integrates into notebooks these days. Some of the stuff that you all are doing and, yeah, it's fascinating. Maybe give people a quick elevator pitch on that while we're on that subject.
On Shiny or on Posit in general? Yeah, whichever you feel like. Yeah, so Posit started out 2009 as RStudio. And so it didn't start out intending to be a company. J.J. Allaire and Joe Chung built a new IDE, Integrated Development Environment for R, because what was available at the time wasn't great. And so they made that into, I think, probably one of the best data science IDEs that's ever been built. It's really an amazing piece of tech.
So it started becoming a company with customers and revenue in the 2013 timeframe. And they've built a whole suite of tools to support enterprise data science teams to make open source data science work in the real world. But the company itself, it's a certified B corporation, has no plans to go public or IPO. It is dedicated to the mission of open source software for data science and technical communication. And basically building itself to be a 100-year company that has a revenue generating enterprise product side and an open source side so that the open source feeds the enterprise part of the business. The enterprise part of the business generates revenue to support the open source development. And the goal is to be able to sustainably support the mission of open source data science for hopefully the rest of our lives.
It is dedicated to the mission of open source software for data science and technical communication. And basically building itself to be a 100-year company that has a revenue generating enterprise product side and an open source side so that the open source feeds the enterprise part of the business.
And it's an amazing company. One of the most successful companies that dedicates a large fraction of its engineering time to open source software development. So it's been very impressed with the company and JJ Allaire, its founder. So I'm excited to be helping it grow and become a sustainable long-term fixture in the ecosystem.
It's definitely doing cool stuff as incentives are aligned well. It's not private equity or IPO. I think it helps.
JJ Allaire and the history behind Posit
Many people know JJ Allaire created ColdFusion, which is the original dynamic web development framework in the 1990s. He and his brother Jeremy and some others built AllaireCorp to commercialize ColdFusion. And they built a successful software business that was acquired by Macromedia, which was eventually acquired by Adobe. But they did go public as AllaireCorp during the dot-com bubble. And JJ went on to found a couple of other successful startups.
I think he found himself in his late 30s 15 years ago or around the age I am now. Having been very successful as an entrepreneur, no need to make money. And looking for a mission to spend the rest of his career on. Identifying data science and statistical computing as an open source, in particular making open source for data science work, was the mission that he aligned with and something that he had been interested in earlier in his career. But he had gotten busy with other things. So I think it's really refreshing to work with people who are really mission-focused and focused on making impact in the world, creating great software, empowering people, increasing accessibility, and making most of it available for free on the internet.
And not being so focused on empire building and producing great profits for venture investors and things like that. So I think the goal of the company is to provide an amazing home for top-tier software developers to work on this software, to spend their careers, and to build families, and to be a happy and healthy culture for working on this type of software.
What is Pandas?
That sounds excellent. I didn't realize the history all the way back to ColdFusion. Speaking of history, let's jump in. Wes, there's a possibility that people out there listening don't know what Pandas is. You would think it's pretty ubiquitous, and I certainly would say that it is, especially in the data science space. But I've got a bunch of listeners who listen, and they say really surprising things. They'll say stuff to me like, Michael, I've been listening for six weeks now, and I'm starting to understand some of the stuff you all are talking about. Why did you listen for six weeks? You didn't know what I was talking about. That's crazy.
A lot of people use it as language immersion to get into the Python space. I'm sure there's plenty of people out there who are immersing themselves but are pretty new. Maybe for that crew, we could introduce what Pandas is then.
Absolutely. It's a data manipulation and analysis toolkit for Python. It's a Python library that you install that enables you to read data files, so read many different types of data files off of disk or off of remote storage or read data out of a database or some other remote data storage system. This is tabular data, so it's structured data like with columns. You can think of it like a spreadsheet or some other tabular data set.
And then it provides you with this data frame object, which is kind of pandas.dataframe, that is the main tabular data object. And it has a ton of methods for accessing, slicing, grabbing subsets of the data, applying functions on it that do filtering and subsetting and selection, as well as more analytical operations like things that you might do with a database system or SQL, so joins and lookups, as well as analytical functions like summary statistics, grouping by some key and producing summary statistics. So it's basically a Swiss army knife for doing data manipulation, data cleaning, and supporting the data analysis workflow.
But it doesn't actually include very much as far as actual statistics or models or if you're doing something with LLMs or linear regression or some type of machine learning, you have to use another library, but pandas is the on-ramp for all of the data into your environment in Python. So when people are building some kind of application that touches data in Python, pandas is often the initial on-ramp for how data gets into Python, where you clean up the data, you regularize it, you get it ready for analysis, and then you feed the clean data into the downstream statistical library or data analysis library that you're using. That whole data wrangling side of things, right? Yeah, that's right.
Python's history with numerical computing
And so it's some history. So Python had arrays like matrices and what we call tensors now, multidimensional arrays going back all the way to 1995, which is pretty early history for Python. The Python programming language has only been around since 1990 or 1991, if my memory serves. But what became NumPy in 2005, 2006 started out as numeric in 1995, and it provided numerical computing, multidimensional arrays, matrices, the kind of stuff that you might do in MATLAB, but it was mainly focused on numerical computing and not with the type of business data sets that you find in database systems, which contain a lot of strings or dates or non-numeric data.
And so my initial interest was I found Python to be a really productive programming language. I really liked writing code in it, writing simple scripts, like doing random things for my job. And you had this numerical computing library, NumPy, which enabled you to work with large numeric arrays and large data sets with a single data type. But working with this more tabular type data, stuff that you would do in Excel or stuff that you do in a database, it wasn't very easy to do that with NumPy or it wasn't really designed for that. And so that's what led to building this higher level library that deals with these tabular data sets in the Pandas library, which was originally focused on, you know, building with a really close relationship with NumPy. So Pandas itself was like a thin layer on top of NumPy originally.
One thing I find interesting about Pandas is it's almost its own programming environment these days in the sense that, you know, traditional Python, we do a lot of loops, we do a lot of attribute dereferencing, function calling, and a lot of what happens in Pandas is more functional, it's more applied to us, it's almost like set operations, right? And a lot of vector operations and so on.
Yeah, and that was behaviors inherited from NumPy. So NumPy is very array oriented, vector oriented. So you, rather than write a for loop, you would write an array expression, which would operate on whole batches of data in a single function call, which is a lot faster because you can drop down into C code and get good performance that way. And so Pandas adopted the NumPy way of, like the NumPy-like array expression or vector operations.
But it's true that that's extended to the types of like non-numeric data operations that you can do in Pandas, like, you know, vectorized set lookups, where you can say like, you would say like, oh, like this, I have this array of strings and I have this subset of strings and I want to compute a Boolean array, which says whether or not each string is contained in this set of strings. And so in Pandas, that's the is in function. So you would say like, you know, column A like is in some set of substrings and that produces, that single function call produces a whole Boolean array that you can use for subsetting later on.
Yeah, there's a ton of things that are really interesting in there. One of the challenges, maybe you could speak to this a little bit, then I want to come back to your performance comment. One of the challenges I think is that some of these operations are not super obvious
that they exist or that they're discoverable, right? Like instead of just indexing into say a column, you can index on an expression that might filter out the columns or project them or things like that. How do you recommend people like kind of discover a little bigger breadth of what they can do?
I mean, there's plenty of, I mean, I think there's great books written about Pandas. So there's my book, Python for Datasets. I think Matt Harrison has written an excellent book, Effective Pandas. The Pandas documentation I think has, provides really nitty gritty detail about how all the different things work.
But when I was writing this book, Python for Data Analysis, my goal was to provide a primer, like a tutorial on how to solve data problems with Pandas. And so for that, I had to introduce some basics of how NumPy works. So people, array oriented computing, basics of Python, so you know enough Python to be able to understand what things that Pandas is doing. But the, yeah, kind of, it builds incrementally. And so like, as you go through the book, the content gets more and more advanced. It introduces, you learn, you master initial set of techniques and then you can start learning about more advanced techniques. So it's definitely a pedagogical, you know, pedagogical resource.
Publishing the book with Quarto
And it is now freely, as you're showing there on the screen, it is freely available on the internet. So JJ Allaire helped me port the book to use Quarto, which is a new technical publishing system for writing books and blogs and website, you know, quarto.org. And there it is.
And yeah, so that's how I was able to publish my book on the internet as, you know, essentially you can use Quarto to write books using notebooks, which is cool. My book was written a long time ago in O'Reilly's doc XML. So not particularly fun to edit, but yeah, but because Quarto is built on Pandoc, which is a sort of markup language transpilation system. So you can use Pandoc to convert from one, you know, you can to convert documents from one format to another. And so that's the kind of the root framework that Quarto is built on for, you know, generating starting with one document format and generating many different types of output formats.
That's cool. I didn't realize your book was available just to read on the internet. Yeah. Yeah. So in the third edition, I was able to negotiate with O'Reilly and make an amendment to my very old book contract from 2011 to let me release the book for free on my website. So it's, yeah, it's just available there at westmckinney.com slash book.
I find that like a lot of people really like the print book. And so I think that having the online book just available, like whenever you are somewhere and you want to look something up is great. But yeah, I've been- Print books are hard to search. Yeah, that's true. That's true. Yeah. And like if you go to the search bar and if you go back to the book and just look at the search bar, you know, just search for like group buy, like all one word or, you know, yeah, it's like it comes up really fast. You can go to that section and it's pretty cool.
I thought that releasing the book for free online would affect sales, but now people just really like having paper books, it seems, even in 2024. Yeah. Even digital books are nice. You got them with you all the time. You can- I think it's about taking the notes. Where do I put my highlights and how do I remember it? That's right. Yeah, yeah. Stuff like that.
Pandoc and the history of Quarto
This quarter thing looks super interesting. If you look at Pandoc, if people haven't looked at this before, the conversion matrix, I don't know how you would, how would you describe this, Wes? Busy? Complete? What is this? This is crazy. It's very busy, yeah. It can convert from, it looks like about, you know, 30 or 40 input formats to, you know, 50 or 60 output formats, maybe more than that. It's kind of my just eyeballing it, but yeah, it's like a pretty impressive.
And then if you took the combinatorial of like how many different ways could you combine the 30 to the 50? It's kind of what it looks like. It's truly amazing. So if you've got Markdown, you want to turn it into a PDF or you've got a Doku wiki and you want to turn it into an EPUB or whatever, right? Or even like RevealJS, probably to PowerPoint, I would imagine. I don't know. Yeah.
Well, as history, like backstory about Quarto, so, you know, it helps to keep in mind that JJ created ColdFusion, which was this, you know, essentially publishing system, early publishing system for the internet, similar to CGI and PHP and other dynamic, you know, web publishing systems. And so early on at RStudio, they created RMarkdown, which is basically extensions to Markdown that allow you to have code cells written in R and then eventually they added support for some other languages where it's kind of like a Jupyter Notebook in the sense you could have some Markdown and some code and some plots and output and you would run the RMarkdown render and it would, you know, generate all the output and insert it into the document. And so you could use that to write blogs and websites and everything.
But RMarkdown was written in R and so that limited, in a sense, like, it made it harder to install because you would have to install R to use it. And also people, you know, it had an association with R that perhaps was, like, unmerited. And so in the meantime, you know, with everything that's happened with web technology, there's, it's now very easy to put a complete JavaScript engine in a small install footprint, you know, on a machine with no dependencies and to be able to run a system that is, you know, written in, a system that's written in JavaScript. And so Quarto is completely language agnostic. It's written in TypeScript and it uses Pandoc as an underlying engine and it's very easy to install and so it addresses some of the portability and, you know, extents that were in RMarkdown.
But as a result, you know, I think our, the Posit team had a lot of, just has more than, you know, a decade or if you include ColdFusion, you know, more than 25 years of experience in building really developer-friendly technical publishing tools. And so I think that it's, it's not data science, but it's something that is an important part of the data science workflow, which is how do you present your, make your analysis and your work available for consumption in different forms. And so having this system that can, you know, publish outputs in many different places is super valuable.
So a lot of people start out in Jupyter Notebooks, but, you know, but there's many different, you know, many different possible input formats. And so to be able to, you know, use the same source to publish to a website or to a confluence page or to a PDF is like super valuable. Yeah, it's super interesting. Okay. Something I got to explore some more.
Pandas' growth and community
All right, let's go back to Pandas for a minute. First, how about some kind words from the audience for you? Ravid says, Wes, your work has changed my life. It's very, very nice. I'm happy to hear it. But yeah, I, yeah, I'm more than happy to, yeah, talk in depth about Pandas. And I think history of the project is interesting. And I think also how the project has developed in the intervening 15, 16 years is pretty interesting as well.
Yeah. Let's talk, let's talk derivatives for a minute. So growth and speed of adoption and all those things. All right. When you first started working on this and you first put it down, did you foresee a world where this was so popular and so important? Did you think of, yeah, pretty soon black holes. I'm pretty sure I'll be part of that somehow.
I mean, it was, it was always, it was always the aspiration. Like there was the aspiration there of making, making Python this mainstream, mainstream language for statistical computing and data analysis. Like I didn't, it didn't occur to me that it would become this popular or that it would become like the, one of the main tools that people use for working with data in a business, in a business setting like that, you know, it would have been, if that was the, if that was the aspiration or if that was, you know what I needed to achieve to be satisfied, that that would have been completely unreasonable.
And I also don't think that, that, you know, in a certain sense, like I don't, I don't know that, that it's popularity, you know, it, I mean, and it's, like I, I think there's, there's many other worthy efforts that have been created over the years that have been the, you know, really, you know, really great work that, that others have, have done in this, this domain. And so the fact that, that Pandas caught on and became as popular as it is, I think it's a combination of timing and, you know, there was like a developer relations aspect that there was content available and like I wrote my book and that made it easier for people to learn about the project.
But also like we, we had a serendipitous open source developer community that came together, that the project to, you know, grow and expand like really rapidly in the, in the early 2010s. And, you know, and I, yeah, I definitely spent a lot of work like recruiting people to, to work on the project and encouraging, you know, others to work on it. Cause sometimes people create open source projects and then it's hard for, hard for others to get involved and get a seat at the table, so to speak. But I was very keen to bring on others and to give them responsibility and, you know, ultimately, you know, hand over the reins to the, to the project, to others.
Cause sometimes people create open source projects and then it's hard for, hard for others to get involved and get a seat at the table, so to speak. But I was very keen to bring on others and to give them responsibility and, you know, ultimately, you know, hand over the reins to the, to the project, to others.
And I've, I've spoken a lot about that, you know, over the years, how important that is to, you know, for open source project creators to room for others in, in growing the project so that they can become, you know, owner, owner of the project. It's tough to make space and tough to, to bring on folks. Have you heard of the Django knots? Django knots. I think it's Django knots dot space. They have an awesome domain, but it's basically like, kind of like a bootcamp, but it's for taking people who just like Django and turn them into actually contributors or core contributors. Do you guys have, what's your onboarding story for people who do want to participate?
the different community outreach channels that the Pandas project has done to help grow new contributors. So one of the core team members, Mark Garcia, has done an amazing job organizing documentation sprints and other contributor sourcing events, essentially creating very friendly accessible events where people who are interested in getting involved in Pandas can meet each other and then assist each other in making their first pull request. And it could be something as simple as, you know, making a small improvement to the Pandas documentation because it's such a large project. The documentation is like always something that could be made better, you know, either adding more examples or documenting things that aren't documented or making, yeah, just making the documentation better. And so it's something that for new contributors is more accessible than working on the internals of like one of the, you know, algorithms or something and, or like we working on some significant performance improvement might be a bit intimidating if you've never worked on the Pandas codebase and it's a pretty large codebase because it's been it's been worked on continuously for, you know, for like going on 20 years. So it's, yeah, it can be takes a while to really get to a place where you can be productive. And that can be discouraging for new contributors, especially those who don't have a lot of open source experience.
That's one of the ironies of the challenges of these big projects is they're just so finely polished. So many people are using them. Every edge case matters to somebody, right? And so to become a contributor and make changes to that, it takes a while, I'm sure.
Yeah. Yeah. I mean, I think it's definitely a big, I think definitely a big thing that helped is allowing people to get paid to work on Pandas or to be able to contribute to Pandas as, as a part of their job description, like as, you know, maybe part of their job is maintaining, maintaining Pandas. So Anaconda, you know, was like, you know, one of the earliest companies who had engineers on staff, you know, like, yeah, like, you know, Brock Mendel, Tom Augsberger, Jeff Reback, who part of their job was maintaining and developing, developing Pandas. And that was, that was huge because prior to that, the project was purely based on volunteers. Like I, I was a volunteer and everyone was working on the project as a, as a, as a passion project in their, in their free time. And then, you know, Travis Oliphant, one of the founders, he and Peter Wang founded Anaconda. Travis spun out from Anaconda to create Quonsight and has continued to, you know, sponsor, sponsor development in and around, in and around Pandas. And that's enabled people like Mark to do these community building events.
and, and for it to not be, you know, something that's, you know, totally, totally uncompensated.
Yeah, that's a lot of, a lot of stuff going on. And I think, I mean, it's, yeah, interest is awesome, right? I mean, if there's just a different level of problems, I feel like you can, you could take on, you know, you know what, I got this entire week and someone that's my job is to make this work rather than I've, I've got two hours and I can't really take on a huge project. And so I'll work on the smaller improvements or whatever.
Yeah. But I, I think, and I think many people know, but I, I haven't been involved day to day in Pandas since 2013. So that's, that's getting on. That's, that's a lot of, that's a lot of years. You know, I still, you know, I still talk to the Pandas contributors. We had a, we had a Pandas meetup, core, core developer meetup here in Nashville, pre COVID, I think it was in 2019, maybe. So, you know, I'm still in active contact with the Pandas developers, but it's, it's been a, you know, a different team of people leading the project. It's taken on a life of its own, which is, which is amazing. And that's exactly, that's, that's as an open, yeah, as a project creator, that's exactly what you want is to not be beholden to the project that you created and forced and, you know, kind of, you know, have to, you know, be responsible for it and take care of it for the rest of the rest of your life.
But if you look at like a lot of the community, a lot of the most kind of intensive community development has happened since, like, since I moved on to work on, on other projects. And so now the project has, I don't know the exact count, but it's had thousands of contributors. And so, you know, to have thousands of different unique individuals contributing to an open source project is it's, it's a big, it's a big deal. So I think even, I don't know what it says on the bottom of GitHub, it says, you know, 30, 3200 contributors, but that's maybe not even the full story because sometimes, you know, people, you know, they don't have their email address associated with their GitHub profile. And, you know, how GitHub counts contributors, I would say probably the true number is closer to 4000. So that's, yeah, but that's, I think that's a testament, you know, to the, to the core team and all the outreach they've done and work making, making the project accessible and easy to contribute to.
Because if people, if you go and try to make a pull request to a project and there's many different ways that you can fail. So, like, either the project is technically, like, there's issues with the build system or the developer tooling. And so you struggle with the developer tooling. And so if you aren't working on it every day and every night, you can't make heads or tails of how the developer tools work. But then there's also, like, the level of accessibility of the core development team. Like, if they don't, if they aren't there to support you in getting involved in the project and learning how it works, and creating documentation, attribute and what's expected of you, that can also be, you know, a source of frustration where people churn out of the project, you know, because it's just, it's too hard to find their sea legs. And maybe, and maybe also, you know, sometimes development teams are unfriendly or unhelpful or, you know, they make others feel like they're annoyed with them or, like, they're wasting their time or something. It's like, I don't want to look at your, you know, this pull request and give you feedback because, you know, I could do it more quickly by myself or something. Like, sometimes you see that in open source projects. But they've created a very, like, they've created a very welcoming environment. And, yeah, I think the contribution numbers speak for themselves.
They definitely do. Maybe last thing before we move on to the other stuff you're working on. But the other interesting GitHub statistic here is the used by 1.6 million projects. That's, I don't know if I've ever seen it used by that high. There's probably some that are higher, but not many. Yeah, it's, yeah, it's a lot of projects.
I think it's interesting. I think it's, like many projects, it's reached a point where it's an essential and assumed part of many people's toolkit. Like, the first thing that they write at the top of a file that they're working on is, you know, import Pandas as PD or import NumPy as PD. And so, it's, yeah, I think, you know, to create, I think, in a sense, like, I think one of the reasons why, you know, Pandas has gotten so popular is that it is beneficial to the community, to the Python community, to have, you know, fewer solutions. You know, kind of the zen of Python, there should be one and preferably only one obvious way to do things. And so, if there were 10 different Pandas-like projects, you know, that creates skill portability problems. And, you know, it's just easier if everyone says, oh, we just, Pandas is the thing that we use. Change jobs and you can take all your skills, like how to use Pandas with you.
And I think that's also one of the reasons why Python has become so successful in the business world is because you can teach somebody, even without a lot of programming experience, how to use Python, how to use Pandas and become productive doing, you know, basic work very quickly. And so, you know, one of the solutions, I remember back in the early 2010s, there were a lot of articles and talks about how to address the data science shortage. And my belief, and I gave a talk at Web Summit in Dublin in 2000, gosh, maybe 2017, I have to look exactly. But basically, it was the data scientist shortage. And my thesis was always, we should make it easier to be a data scientist or lower the bar for what sort of skills you have to master before you can do productive work in a business setting. And so, I think the fact that there is just Pandas and that's the one thing that people have to learn how to use is their essential starting point for doing any data has also led to this piling on of people being motivated to make this one thing better because it, you know, you make improvements to Pandas and they benefit millions of projects and millions of people around the world. And that's, yeah. So, it's like a, you know, steady snowballing effect.
my thesis was always, we should make it easier to be a data scientist or lower the bar for what sort of skills you have to master before you can do productive work in a business setting.
WebAssembly and the future of data tools
I agree with you. And I think that, I think doing data science is getting easier. We've got a lot of interesting frameworks and tools for Python, one of them, right? That makes it easier to share and run your code, you know? Yeah. So, yeah. I think the, yeah, Shiny for Python, Streamlit, you know, Dash, like these different interactive data application publishing frameworks. So, you can go from, you know, a few lines of Pandas code, loading some data and doing some analysis and visualization to changing that as an interactive website without having to know how to use any web development frameworks or Node.js or anything like that. And so, to be able to, you know, get up and running and build a working, you know, interactive web application that's powered by Python. Yeah. It's a game changer in terms of, you know, shortening end-to-end development life cycles.
What do you think about JupyterLite and these PyOxidized and basically Jupyter in a browser type of things?
Yeah. WebAssembly and all that. Yeah. So, I'm definitely very excited about it. I've been following WebAssembly in general. And so, I guess some people listening will know about WebAssembly. But basically, it's a portable machine code that can be compiled and executed in within your browser in a sandbox environment. So, it protects against security issues and allows prevents, like, the person who wrote the WebAssembly code from doing something malicious on your machine, which is very important. Won't necessarily stop them from, like, you know, mining cryptocurrency while you have the browser tab open. That's a whole separate problem.
But, yeah, I think it's enabled us to run the whole scientific Python stack, including Jupyter and NumPy and Pandas totally in the browser without having a client and server and needing to, you know, run a container someplace in the cloud. And so, I think in terms of creating application deployment, so, like, being able to deploy an interactive data application, like, with Shiny, for example, without needing to have a server, that's actually pretty amazing. And so, I think that, you know, simplifies and opens up new use cases, like, new, you know, application architectures and makes things a lot easier. Because setting up and running a server creates brittleness, like, it has cost. And so, if the browser is doubling as your server process, like, that's I think that's really cool.
You also have, like, other projects, like DuckDB, which is a, you know, high performance, embeddable analytic SQL engine. And so, you know, now with DuckDB compiled to Wasm, you can get a high performance database running in your browser. And so, you can get low latency interactive queries and interactive dashboards. And so, it's yeah, there's it's WebAssembly has opened up this whole kind of new world of possibilities. And so, it's transformative, I think.
I think for Python in particular, you mentioned Pyodide, which is kind of a whole package stack. So, it's like a framework for build and build and packaging and, you know, basically building an application and managing its dependencies. So, you could create a WebAssembly version of your application to be deployed like this. But yeah. So, I think the one of the Pyodide either the Pyodide main creator or maintainer went to Anaconda. They created PyScript, which made which is another attempt to make it even easier to use Python to make it even easier to use Python to create web applications, interactive web applications. And yeah. So, there's so many cool things here. Like in the R community, they have WebR, which is similar to PyScript and Pyodide in some ways, like compiling the whole R stack to WebAssembly. There was just an article I saw on Hacker News where they worked on, you know, figuring out how to get how to trick LLVM into compiling Fortran code, like legacy Fortran code to WebAssembly. Because when you're talking about all this, you know, scientific computing stack, you need the linear algebra and all of the, you know, 40 years of Fortran code that have been built to support scientific applications, like you need all that to compile to and run in the browser. So.
Yeah. That's pretty wild to think of putting that in there, but very useful. I didn't realize that you could use DuckDB as a WebAssembly component. That's pretty cool. Yeah. Like there's a company, I'm not an investor or plugging them or anything, but it's called evidence.dev. And it's like a whole business intelligence, open source business intelligence application that's powered by DuckDB. And so if you have data that fits in the browser, you know, to have a whole like interactive dashboard, or to be able to do business intelligence, like fully in the browser with no need of a server. It's, yeah, it's, it's, it's, it's very, very cool.
So I'm, yeah, I've been following DuckDB since the early days. And, you know, my company, Voltron Data, like, you know, we, we became members of the DuckDB foundation and built, you know, build, actively build a relationship with with DuckDB labs, so we could help accelerate progress in this space, because I think the impact, the impact is so, is so immense. And we just, we, it's hard to predict, like what, you know, what people are going to build, build with all this stuff. And, and so that was all, you know, with, I guess, going back, you know, 15 years ago to Python, like one of the reasons I became so passionate about building stuff for Python was about in, I think, the way that Peter Wang puts that, it puts it as, you know, giving people superpowers. So we want to enable people to build things with much less code and much less time. And so by making it things that much more accessible, that much easier to do, like in the mantra in pandas was like, how do we make things, you know, one line of code, or like, this must be easy. It's like, one line of code, one line of code, it must be like, like, make this as terse and simple as possible so that you can move on and focus on building the more interesting parts of your application rather than struggling with how to read a CSV file or, you know, how to do, you know, whichever, whichever, you know, data munging technique that you need for your, for your data set.
the way that Peter Wang puts that, it puts it as, you know, giving people superpowers. So we want to enable people to build things with much less code and much less time.
DuckDB and SQLite comparison
Maybe an interesting mental model for DuckDB is kind of an equivalent to SQLite, but more analytics database for folks, you know, in process. Yeah, things right. What do you think? Yeah. So yeah, DuckDB is like SQLite. And in fact, it can run the whole SQLite test suite, I believe. So it's a full database, but it's, it's, it's for analytic processing. So it's optimized for analytic processing. And as compared, you know, with SQLite, which is not, it's, it's not designed for, for data processing.
Yeah, cool. All right. Well, let's talk about some things that you're working on beyond pandas. You talked about Apache Aero earlier. What's, what are you doing with Aero and, and how's it fit in your world? Yeah. So the, the, the backstory there was,
I don't know if you can hear the sirens in, in, in downtown Nashville, but... No, actually it's... That's good. It's filled, the microphone filters it out, filters it out pretty well. Yay for dynamic microphones. They're amazing.
Origins of Apache Arrow
Yeah. So in, like around the mid, like the mid 2010s, 2015, I, I started working, I started working at Cloudera, like in the, which is a company that was like one of the pioneers in the big data ecosystem, big data ecosystem. And yeah, I had, you know, been, spent several years working on five, five years, five, six years working on pandas. And so I had gone through the experience of building pandas from top to bottom. And it was this, you know, full stack system that had, had its own, you know, mini query engine, all of its own algorithms and data structures and all this stuff that we had to build from, build from scratch.
And I started thinking about, you know, what if it was possible to build some of the underlying computing technology, like data readers, like file readers all the algorithms that power the core components of pandas, like group operations, aggregations, filtering selection, all those things. Like what if they were possible to have a general purpose library that isn't specific to Python is, but it was really, really fast, really efficient and has a large community building, building it so that you could, you know, take that code with you and use it to build many different types of libraries, not just data frame libraries, but also database engines and, you know, stream processing engines and all kinds of things. And so that was kind of what was in my mind when I started getting interested in what turned into Arrow.
And one of the problems we realized we needed to solve, this was like a group of other open source developers and me, was that we needed to create a way to represent data that was not tied to a specific programming language. And that could be used for a very efficient interchange between components. And the idea is that you would have this immutable, like this, this kind of constant data structure, which is like, it's the same in every programming language. And then you can use that as the basis for writing all of your algorithms. So as long as it's Arrow, you have these reusable algorithms that process, process Arrow data. So we started with building the Arrow format and standardizing it. And then we've built a whole ecosystem of components, like library components and different programming languages for building applications that use the, use the Arrow format.
So that includes, you know, not only tools for building and interacting with the data, but also file readers. So you can read CSV files and JSON data and Parquet files, read data out of database systems, you know, wherever the data comes from, we want to have an efficient way to get it into the Arrow format. And then we moved on to building data processing engines that are native to the Arrow format, so that Arrow goes in, the data is processed, Arrow goes out. So DuckDB, for example, supports Arrow as a preferred input format and is, DuckDB is more or less Arrow-like in its internals. It has kind of Arrow, Arrow format plus a number of extensions that are DuckDB specific for better performance within the context of, of DuckDB.
And so, so in numerous communities, so there's the Rust community, which has built DataFusion, which is an execution engine for Arrow, SQL engine for Arrow. And so, yeah, we've kind of like looked at the different layers of the stack, like data access, computing, data transport, you know, everything under the sun. And then we've built libraries that are across many different programming languages said that, that are, you can pick and choose the pieces that you need to build your, build your system. And the goal ultimately was that we, in the future, which is now, we don't want people to have to reinvent the wheel whenever they're building something like Pandas, that they could just pick up these off the shelf components. They can design the developer experience, the user experience that they want to create. And, and they can get built, you know, so if you were building Pandas now, you could build a Pandas-like library based on the Arrow components in much less time. And it would be, it would be fast and efficient and interoperable with the whole ecosystem of other projects that use Arrow.
So it's, it's, it's very cool. It's, I mean, it was really ambitious. In some ways, obvious to people, they would, they would hear about Arrow and they say, that sounds obvious. Like clearly we should have a universal way of transporting data between systems and processing it in memory. Why hasn't this been done in the past? And it turns out that as is true, many open source software problems that, that many of these problems are, the social problems are harder than the technical problems. And so if you can solve the, the kind of people coordination and consensus problems, solving the technical issues is much, much easier by comparison.
And so if you can solve the, the kind of people coordination and consensus problems, solving the technical issues is much, much easier by comparison.
So I think we were lucky in that we found like the right, you know, the right group of people, the right personalities where we were able to, you know, as soon as I met, I met Jacques Nadeau, who had been at MapR and we, you know, was working on his, his startup Dremio. Like I knew instantly when I met Jacques Nadeau, I was like, it's like, I can work. He's like, it's like him, like, he is going to help me make this happen. When I met Julien Ledam, who had also co-created Parquet, I was like, yes, like we are going to make this, like, like we, like I found the right people, like we are, we are going to make this happen. So it's been a, it's been a labor of love and, and, and, and much, much work and, and stress and everything. But it's yeah, that, that's, you know, I've been working on things circling, you know, with Aero as the sun, you know, I've been building kind of satellites and moons and planets circling the, the, the Aero sun over the last, over the last eight years or so. And that's kept me pretty busy.
Arrow and modern hardware efficiency
Yeah. It's only getting more exciting and interesting. Over here, it says, it uses efficient analytic operations on modern hardware, like CPUs and GPUs. You know, one of the big challenges of Python has been the GIL, also one of its big benefits, but one of its challenges when you get to multicore or computational stuff has been the GIL. What's the story here? Yeah. So in, in, in AeroLand, when, when we're talking about analytic efficiency it, it mainly has to do with the like underlying, like how the, how a modern CPU works or how, how, how a GPU works. And so when the data is arranged in column oriented format, that enables the, the data to be moved efficiently through the CPU cache pipelines. So the data is made, made available efficiently to the, to the CPU cores.
And so we spent a lot of energy in Aero making decisions, firstly, to enable very cache, like CPU cache or GPU cache efficient analytics on, on the data. And so, so we were kind of always, when we were deciding, we would break ties and make decisions based on like, what's going to be more efficient for the, for the computer chip. The other thing is that modern, and this is true with GPUs, which have a different parallelism model than, or a different kind of multicore parallelism model than CPUs. But in CPUs, they've focused on adding what are called single instruction, multiple data intrinsic, like a built-in operations in the processor where, you know, now you can process up to 512 bytes of data in a single CPU instruction. And so that's like, you know, my brain's doing the math, right? Like 16, 32 bit floats, or, you know, eight 64 bit integers in a single cycle. There's like intrinsic operations.
Instead of multiply this number by that one, multiply that number to these eight things all at once, something like that. That's right. Yeah. Or you might say like, oh, I, I have a bit mask and I want to select, I want to gather like the, the, the one bits that are set in this, you know, this bit mask from this array of, of integers. And so there's like a gather instruction. which allows you to select a subset, a sort of SIMD vector of integers, you know, using a bit mask. And so that, you know, turns out to be like a pretty, you know, pretty critical operation in certain data analytic workloads.
So yeah, we were really, you know, we wanted to have a data format that was essentially, you know, future-proofed in the sense that it's ideal for the coming wave, like current generation of CPUs, but also given that a lot of processing is moving to GPUs and to FPGAs and to custom silicon, like we wanted Arrow to be usable there as well. And Arrow's been successfully, you know, used as the foundation of GPU computing libraries, like we, you know, kind of at Voltron Data we've built a whole accelerator-native, GPU-native, you know, scalable execution engine that's Arrow-based. And so I think the fact that we, that was our aspiration, and we've been able to prove that out in real-world workloads and show the kinds of efficiency gains that you can get with using modern computing hardware correctly, or at least as well as it's intended to be used. That's a big deal in terms of like making applications faster, reducing the carbon footprint of large-scale data workloads, things like that.
Ibis and the multi-engine data stack
Yeah. Amazing. All right. Let's see. What else have I got on deck here to talk to you about? Let's, maybe we can talk about, we got a little time left. We got a couple of things to cover. Yeah, we can talk about this. Yeah, we could probably spend another hour talking. Yes, easy.
I think one of the more interesting areas in recent years has been new DataFrame libraries and DataFrame APIs that transpile or compile to different, execute on different backends. And so around the time that I was helping start Arrow, I created this project called Ibis, which is basically a portable DataFrame API that knows how to generate SQL queries and compile to pandas and polars and different DataFrame backends. And the goal is to provide a really productive DataFrame API that gives you portability across different execution backends with the goal of enabling what we call the multi-engine data stack. So you aren't stuck with using one particular system because all of the code that you've written is specialized to that system. You have this tool, so maybe you could work with DuckDB on your laptop or pandas or polars with Ibis on your laptop. But if you need to run that workload someplace else, maybe with ClickHouse or BigQuery, or maybe it's a large big data workload that's too big to fit on your laptop, and you need to use Spark SQL or something that you can just ask Ibis, say, hey, I want to do the same thing on this larger data set over here, and it has all the logic to generate the correct query representation and run that workload for you. So it's super useful.
But there's a whole wave of work right now to help enable people to work in a pandas-like way, but work with big data or get better performance than pandas, because pandas is a Swiss army knife, but isn't a chainsaw. So if you were rebuilding pandas from scratch, it would end up a lot. There's areas of the project that are more bloated or have performance overhead that's hard to get rid of. And so that's why you have Richie Fink started the Polars project, which is kind of a reimagining of pandas data frames written in Rust and exposed in Python. And Polars, of course, is built on Apache Aero at its core. So building an Aero-native data frame library in Rust and all the benefits that come with building Python extensions in Rust. You avoid the gill, and you can manage the multithreading in a systems language, all that fun stuff.
Yeah, when you're talking about Aero and supporting different ways of using it and things being built on it, certainly Polars came to mind for me. You know, when you talk about Ibis, I think it's interesting that a lot of these data frame libraries, they try to base their API to be pandas-like, but not identical, potentially, you know, thinking of Dask and others. But this Ibis sort of has the ability to configure it and extend it, make it different, kind of like, for example, Dask, which is one of the backends here. Yeah, it does. But the API doesn't change, right? It just, it talks to the different backends.
Yeah, there's different schools of thought on this. So there's another project called Modin, which is similar to Ibis in many ways, in the sense of, like, transpilation and sort of dynamically supporting different backends, but sought to closely emulate the, you know, exact details of, like, you know, the API call, the function name, the function arguments must be exactly the same as pandas, with the goal of being a drop-in replacement for people's pandas code. And that's one approach, kind of the pandas emulation route. And there's a library called Koalas for Spark, which is like a PySpark emulation layer for the pandas API. And then there's other projects like Polars and Ibis and Dask DataFrame that take, like, design cues from pandas in the sense of, like, the general way in which the API works, but has made meaningful departures in the interest of doing things better in many ways than pandas did in certain parts of the API, and making things simpler and not being beholden to decisions that
were made in pandas, you know, 15 years ago. Not to say there's anything bad about the pandas API, but, like, with any API, it's large. Like, it's very large, as evidenced by, you know, the 2,000 pages of documentation. And so I understand the desire to make things simpler while also refining certain things, making certain types of workloads easier to express.
And so Polars, for example, is very expression-based. And so everything is column expressions and is lazy and not eagerly computed, whereas pandas is eager execution, just like NumPy is, which is how pandas, you know, became eagerly executed in the first place. And so I think the mantra with Polars was, we don't want to support the eager execution by default that pandas provides. We want to be able to build expressions so that we can do query optimization and take inefficient code and under the hood rewrite it to, you know, be more efficient, which is, you know, what you can do with a query optimizer.
And so ultimately, like, and that matters a lot when you're executing code remotely or in, like, a big data system, that you want to have the freedom to be able to take, like, a lazy, you know, analytic expression and rewrite it, you know, based on it might be, like, you need to seriously rewrite the expression in the case of, like, Dask, for example. Like, Dask has to do planning across a distributed cluster. And so, you know, Dask DataFrame is very pandas-like, but it also includes some explicit details of being able to control how the data is partitioned and being able to have some knobs to turn in terms of, like, having more control over what's happening on a distributed cluster. And I think the goal there is, like, to give the developer more control as opposed to, like, trying to be intelligent, you know, make all of the decisions on behalf of the developer.
So if you, you know, if you know about how, you know, know a lot about your data set, then you can make more, you can make, you know, decisions about how to schedule and execute it. Of course, Dask is building, you know, query optimization to start making more of those decisions on behalf of the user. But, you know, Dask has become very popular and impactful in making distributed computing easier in Python. So they've gotten, you know, I think gotten a long way without turning into a database. And I think Dask never aspired to be a database engine, which is a lot of distributed computing is, you know, not database-like. It could be distributed array computing or distributed model training and just being able to easily run distributed, you know, Python functions on a cluster and do distributed computing that way.
It was amazing, like, how many people were using PySpark in the early days just for the convenience of being able to run Python functions in parallel on a cluster. Yeah. And that's pretty interesting. Not exactly what it's designed for. Right. With Dask, you know, you probably come across situations where you do a sequence of operations. They're kind of commutative in the end in practice, but from a computational perspective, like how do I distribute this amongst different servers? Maybe one order matters a lot more than the other out of performance, you know?
SQLglot and SQL dialect transpilation
Yeah. Interesting. All right. One final thing. SQLglot.
Yeah. So SQLglot project started by Toby Mao. So he's a Netflix alum and, you know, really, yeah, really talented, talented developer who's created this SQL query transpilation framework library for Python and, you know, kind of underlying core library. And so the problem that's being solved there is that SQL, despite being a quote unquote standard, is not at all standardized across different database systems. And so if you want to take your SQL queries written for one engine and use them someplace else, without something like SQLglot, you would have to manually rewrite and make sure you get the typecasting and coalescing rules correct.
And so SQLglot understands the intricacies and the quirks of every database dialect, SQL dialect, and knows how to correctly translate from one dialect to another. And so Ibis now uses SQLglot as its underlying engine for query transpilation and generating SQL outputs. So originally Ibis had its own kind of bad version of SQLglot, kind of a query like SQL transpilation that was powered by I think powered by SQLalchemy and a bunch of custom code. And so I think they've been able to delete a lot in Ibis by moving to SQLglot. And I know that, you know, SQLglot is also, you know, being used to, you know, power kind of a new, yeah, being used in people building new products that are Python powered and things like that. So, and Toby, like his company, Tobiko Data, they, yeah, they're building a product called SQLmesh that's powered by SQLglot. So very cool project and maybe a bit in the weeds, but if you've ever needed to convert a SQL query from one dialect to another, it's, yeah, SQLglot is here to save the day.
if you've ever needed to convert a SQL query from one dialect to another, it's, yeah, SQLglot is here to save the day.
I would say, you know, even simple things is how do you specify a parameter variable, you know, for a parameterized query, right? And Microsoft SQL Server, it's like at the parameter name and Oracle, it's like question mark or SQLite, I think it's also, you know, just that, even those simple things. It's a pain. Yeah. And without it, you end up with little bobby tables, which is also not good. So that's true. That's true. Nobody wants to talk to him. Yeah. This is really cool. SQLglot, like polyglot, but all the languages of SQL.
Nice. And you do things like, you can say, read DuckDB and write to Hive or read DuckDB and then write to Spark or whatever. It's pretty cool. Yeah.
Wrapping up
All right, Wes, I think we're getting short on time, but, you know, I know everybody appreciated hearing from you and hearing what you're up to these days. Anything you want to add before we wrap up? I don't think so. Yeah. I enjoyed the conversation and yeah. Yeah, there's a lot of stuff, a lot of stuff going on and still plenty of things to get excited about. So I think often people feel like, you know, all the exciting problems in the Python ecosystem have been solved, but there's still a lot to do. And yeah, we've made a lot of progress in the last 15 plus years, but, you know, in some ways it feels like we're just getting started. So excited to see where things go next.
Yeah. Every time I think all the problems are solved, then you just have all these new things that are so creative and you're like, oh, that was a big problem. I didn't even know it was a problem. It's great. Yeah. Well, all right. Thank you for being here and taking the time and yeah. Keep us updated on what you're up to. Sounds good. All right.