Resources
Jan 27, 2026 |
1h 1min
|
1.1k views

Emily Riederer: Column selectors, data quality, and learning in public

Emily Riederer writes Python with an R accent, and we’re all comfortable with it. In this episode, Emily reflects on her journey through R, Python, and SQL — from lessons learned in averaging default values (oops, we're not all rich!) to discovering that column selectors are way cooler than they sound. She weighs in on the delicate art of learning in public, why frustration often makes the best teacher, and how to find your niche by solving the boring problems. Oh, Oh, and the crew casually drops that she's keynoting posit::conf(2026)!

Emily’s had a wild ride through modeling, data engineering, machine learning, and back again, and she knows a thing or three about the evolution of SQL tooling (from nightmare multi-page scripts to the dbt renaissance). She reveals how building internal packages became her gateway to making work enjoyable. Plus: the surprising Stata origins of column selectors, the eternal struggle of naming packages across R and Python, and why watching people code teaches you more than any tutorial ever could. The conversation gets real about imposter syndrome and the magic of tacit knowledge.

IN THIS EPISODE
• Why real-world data is chaos, not truth
• The path from modeling to data engineering (and back)
• What a data pipeline really is (extract, load, transform) and why organization matters
• How dbt changed the SQL game
• Learning by watching: Tacit knowledge and coding over the shoulder
• Imposter syndrome and learning in public
• Building internal tools to escape busywork
• posit::conf(2026) keynote preview

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Welcome to the test set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.

On this episode, we're joined by Emily Riederer, who I think has the distinction of living in that sweet, sweet overlap between Python, SQL, and R harder than anyone else I've ever seen, and is a data science manager at Capital One, and apparently listens to the test set, but is not planning to listen to this episode. Welcome to the test set. I'm Michael Chow, and I'm joined by my co-hosts, Wes McKinney, who's a principal architect at Posit, and Hadley Wickham, who's a chief scientist at Posit. And I'm so excited to be here with Emily Riederer, who is a data science manager at Capital One, and I think a sort of like icon in the R, Python, SQL community for just putting out so much interesting work in this intersection of Python, SQL, and R.

So like, recently the article Python Ergonomics, and some talks around that, of how to have a workflow that R users love in Python with things like pollers, and dbtplyr, which was a plugin for the SQL framework dbt, which was a really interesting sort of cross-section of ideas. So Emily, thanks for coming on. So happy to have you.

Oh yeah, thank you so much for having me. I've loved the pod so much so far. I have not missed a single episode. Yo, I'm so glad. Honestly, the biggest downside of being on a podcast is I was thinking about it, and I'm like, oh, the next one that comes out, I'm not going to want to listen to. Yeah, are you going to listen to your own? I feel like, oh my goodness, you can't break the streak, you know? Never, never, never, never.

So I think people will be really interested to hear sort of your journey through languages like R, Python, SQL, and the things you've put out.

Emily's journey into SQL

But one interesting question I thought might be good to open up with is maybe just on the role of SQL kind of over the years. Since I know you started doing a lot of R work, and folks here have done like a lot of work in SQL, whether it's like dbtplyr, which translates like dplyr and R to SQL, or ibis, which translates Python to SQL. Emily, I'm really curious to hear like, what was your journey into SQL like?

Very abrupt in some ways, because I think if I think about my educational background, I was in a relatively like theoretical stats program, so could prove like asymptotic convergence of a lot of things, had very little experience with real world data, until I get one of my first internships, I'm asked to make a customer profile, I take the average of customer incomes, and I'm like, we're all rich, but we were not rich, I was just averaging in all the 9999 encoded default values. So that was my first introduction, both to databases, to SQL, and just like the vagaries of real world data. And I think that had been like, was also kind of jarring, because I was under this whole illusion of like, oh, data is this like, ground truth, where we can like figure things out about the world. And then I entered into this world where like, data is this like, kind of source of chaos that you need to control.

And then I entered into this world where like, data is this like, kind of source of chaos that you need to control.

But then kind of like, kind of coming into industry, I think SQL was one of like, the main tools for the trade, the only way to kind of like, access your data before you could get in there with something like R or Python. And pulling them from just like kind of a single brush I'd had in college with kind of database design and data modeling and verbal forms, I think just something about that, combined with the tidyverse, just so kind of like, clicked in my mind of like, there's actually a real art to how you like, set this thing up, how you like, get the data out, that can really just like, set you up for success.

Yeah, it feels so real that like, moving into a business, like going to work, and hitting like SQL, and realizing that there's like a process to get the data out. Yeah, I'm curious your experiences with SQL too, where you both kind of, it seems like ramped up your SQL involvement at some point.

I mean, I will say I've been in an active like, I think calling it a war against SQL is maybe putting a little bit strongly, but I've definitely engaged heavily, and I think as has Hadley, in building tools so that humans write a lot less SQL. And I've often felt that SQL is as a language, it's clearly not going anywhere, it's the lingua franca of databases. But it's also a little bit like assembly code or Fortran code, it doesn't have a lot of the modern niceties of real, like a real programming language, you know, like functions, like reusable code, like code that you can refactor and reuse. And so, like early on in my career, I was a bit scarred by hundreds and thousands of lines of copy and pasted SQL queries that, you know, there's no tests, and so, you know, you're dealing with these like highly brittle, very complex, you know, hundreds of lines, many pages long, 10 page long, SQL queries. And, you know, the reality is like SQL is really alluring in the sense that like, it's declarative, you know, writing simple SQL queries is easy, but complex business logic, especially in financial setting ends up being rather subtle and with a lot of complexities. And so, I found myself like making the same kinds of mistakes over and over again and seeing other people make mistakes. And so, I felt like if only, you know, if we could essentially abstract away the unpleasantness of SQL and make it easier for humans to write, essentially author SQL indirectly and to avoid many of those common errors, then that would be, we would be doing humanity a great service.

I guess I actually started with databases, like I did SQL before R. So, my dad, a lot of his work involved databases. So, we had like, you know, we had like dinner table conversations about like relational data and COD's third normal form. And when I was in high school, I guess starting from like age, I don't know, like 15, I like made XS databases as like my part-time job. I did some like database documentation as a part-time job, which is kind of like crazy looking back at it now. Like I was, I mean, that's really how I learned to program was in like Visual Basic for applications. Like that was my first real exposure to programming, real exposure to SQL. And kind of interestingly, like I've been working on dbplyr lately, which translates dplyr code into SQL. And people, like dbplyr has an access backend and people file issues that it doesn't work. And the fact that like, that is mind-blowing to me that even in like 2025, people are writing R code to connect to a Microsoft XS database and like work with that data. That, I don't know, that's just kind of like mind-blowing, mind-blowing to me. Yeah, that's fair. That's, that does feel like the business reality that someone has like an access database somewhere that they need to wrangle and it's like surfacing in these tools.

Becoming a data scientist

Yeah, I'm so curious, like, so your, Emily, like your journey into data science and how you encountered a lot of these things. Could you, do you mind explaining a little bit just how you became a data scientist? Like what did that journey look like for you? I mean, in some ways, like I think I have like a very boring or traditional data science background in some ways, but in like, I think there's a different read where it's like kind of funny because at every step of the way, I knew what I wanted to do. I probably didn't know the right reason why I wanted to do it. So kind of started out like in college or in high school, took my first stats class, just like had been a math, like kind of math kid figuring out what can I do with math? And just had this like idea of this being like this amazing truth-seeking like applied way to like do math in the real world, I think. And then over time, kind of like there's all this hype around big data and data science and it seemed like a natural progression into the field.

I think, I mean, definitely going in, I didn't actually know probably like so much of the things that were true about it. The data science is like so much more of an art than a science, that it requires so much more engineering skills and you'll like never once again in your life feel the certainty of math, which is probably the thing I liked about math in the first place. But yeah, like I think in some ways like very linear journey, but it's kind of funny to me looking back on how completely on false pretenses.

I think I saw you mentioned like your first, your early work was more around like modeling and using R for modeling, is that? In my time at Capital One, my current employer, I've really worn like three very different hats, which also like maybe kind of mirrors some of the different tools in the data space. Started out working a lot more on problems of like measurement, causal inference, understanding, you know, the values of different like levers you could pull and customer lifetime values. And that's like a lot more exploratory type work, you know, a lot of more like visualization and modeling and just like being a lot more intimate with kind of like both the data and the business.

Then kind of like took probably a sharp right turn to move upstream and spent like a number of years in more of the tools and data stack. So I'm thinking about building out data pipelines or in Python tooling for kind of a broader community to use. And really just like spent a lot more time thinking about how does good coding practices, how does automation, how does good engineering really enable better analytics before moving back into the like core traditional like machine learning modeling type space.

The shift to data engineering

I feel like the switch from like the modeling to the data engineering, it's so intriguing. Like what was that like? Like what were the sort of tools you switched into and things you used? In some ways I like fell into that one pretty like organically even though the like actual roles seem on paper very different. I think in part for what we were talking about of like you joined a company and suddenly like you find out like in school the data sets you were working with was penguins or I'm old enough I'll date myself and say iris. It just like embedded in some nice little toy data set. But you know, I mean so much of your work even if you want to do that really like exploratory deep analytical work is around getting the data. And I think I both had kind of Wes's reaction of like I'm working on this like huge long multi-page SQL script and they're all these subqueries and nested tables and I'm trying to draw it out on a board like the Always Sunny in Philadelphia map.

But at the same time like there was something to me that was like so interesting about both how like and like I think I was feeling all the pain points so closely that like I can't focus on the thing I want until we get data right. So starting to get like obsessed with kind of like column names, data quality checks, how can you actually do like testing and macros and all the things Wes called out that aren't native to kind of SQL. So it's like spending a lot of my time outside of work trying to like understand build out that part as well as realizing like truly understand data. We talk about from a statistics perspective like understanding the data generating process but there's like a separate data generating process that's like the data pipeline process that is like the number one thing that predicts kind of what the data areas will be, what the failure cases are. So I think just kind of like fell in love in service of like understanding the problem in the last mile with thinking about the data. And then at the same time the thing I was working on without going into too much detail I found myself kind of like repeating for many different use cases in a way that just like felt so anathema to me to just like how I saw people in like the RStats Twitter world outside of work like building packages, sharing code, like easier to collaborate with someone in Australia than collaborate with someone in the same company sometimes. And it was like that seems strong which is how I started like getting into like internal package development, tool building. So just like kind of thought I was doing it all in service of the analytical space but more and more I found my like time and energy like gravitated towards these upstream problems.

Breaking down the data pipeline

And if you had to like break down for yourself like at that time like what a data pipeline is and like what are kind of the the key pieces involved like how would you kind of break that down for someone? I mean the most kind of I guess classic go-to paradigm that you can think of with that is something like extract, load, transform. And even probably before extract there's the step we don't talk about a lot that's like logging or encoding of like somehow something that happens in the real world has to be turned into some digital signal. Then extract being someone has to go be like capturing that signal and getting all the different data sets into one sort of centralized place in some sort of format. Depending on your field maybe that's hitting up APIs, maybe that's working with vendors, maybe that's even being a field scientist and doing manual like work in a notebook and then function it into a computer. But then once that's kind of like loaded into a data lake or a data warehouse you still need to impose some level of like organization which at a high level you can think about as maybe being more like organizing the files in your file system. But there are actually different conventions around that if you're using blob storage like S3 which is more like literally organizing files on your hard drive or doing it in a database where in some ways there are a lot more rules and constraints but that helps you get a lot more stuff for free like discoverability, some level of kind of like constraints and checks on internal integrity. All in service of getting your data kind of structured in a format that it's both easy to maintain meaning easy to add, update, change to reflect the real world and also easy for your

downstream users to understand and use. And is this the point where you started using like dbt more or in some of this stuff or? Down the road I think a long time but like I think dbt was like still really in its infancy at the point I started doing a lot of this.

But this is definitely the point where I think started feeling very very acutely on a daily basis the exact sort of pain points that dbt solves for. That like I could build kind of my own workflows for like oh how do I like if I want to test some sql code maybe not harder to set up test cases and unit tests not have a need to do that. I had some like crazy workflows like pulling data down into an rmarkdown notebook from my database and you know kind of like doing a lot more round trips than necessary. Figuring out a big part of data pipelines is orchestration which like I don't think you run into so much in analysis but if you have like a lot of long-running things especially that depend on systems that aren't in your control thinking about how do you make sure all of those everything like happens in the right order. Which is not a hard problem there are a lot of great open source tools for doing that but maybe not always ones analysts have at their disposal and fingertips to spin up the right infrastructure for.

So I think like I just like had my elevator pitch for like the seven or ten things that were the biggest pain points of sql and then one day I was like just at the gym listening to like a data engineering podcast and heard one of the first like interviews or I don't say first to me interviews about dbt and I was just like oh wow they like these people happen to like be interested in solving like some of the exact same problems I've been thinking of. Which I think like for a certain time in the data space was like a weirdly frequent feeling of just that sense of serendipity where like you'd be working about something or thinking about something and then within a week somebody like put out a dbt put out an R package ask the exact same question on stack overflow and it was definitely just like kind of serendipitously fell into it that I was having the same problem as a lot of other people.

Yeah I found like when I fit like it took me a while to understand like what dbt was like what was the like why were people so excited about it until I kind of realized it's like it's like an IDE for SQL plus the ability to write functions plus the ability to use like get and I was like oh wow that's like you'd like you kind of you didn't have that already for SQL that's like crazy and obviously people are going to like love it now because I love this tool because how do you do like software engineering without like functions and version control.

Like yeah I've realized it's like for that reason a very hard thing to make kind of the elevator pitch for because it's you know I mean if you turn somebody that like just has a couple of queries they're running and they're like hey what if you broke this into like 27 queries and added some yaml files in there and they're like oh my goodness that sounds like terrible but once it's like once you've like I think it's like super easy to sell to like people that have like lived in the old world but then people coming into it now like I know there's a lot of talk of like dbt versus sqlglod and like anything that you take is like the base case it's like you can see like oh wow there's still a lot of room for improvement here. Yeah if you like explain the pitch for dbt it might actually sound kind of like more complicated like it's quite a bit you're asking the person to do so if the person's already doing a little bit of sql it might be kind of like a turn off but to have these point like all these things then bring like testing and version control and kind of like engineering practice and there's other things like like lineage and like other yeah essentially stuff that provides insights and intelligence into into data pipelines like otherwise you'd be essentially flying without flying without instrumentation of course now it's funny because like there at one point this whole whole ecosystem seemed very fragmented and now it seems that everything is at phytran so hopefully they'll you know now that they're all almost many of the the folks like sqlglod dbt um you know all you know sdf they were building the uh you know intelligence for data engineering and there are they were required by acquired by dbt so all the folks are working together so hopefully that will i hope that that will that will yield a more integrated and stack of tools that play more nicely together and the people are people are happy to use but i've met a lot of these folks and they they seem like they're they're um they're you know they're they're all working to to for the benefit of the actual users and making things better so i'm hopeful.

The other thing i find exciting is trying to see like a little bit of innovation in the sql space like i think dark db has like introduced a number of like sql extensions we are like like oh my god like why has no one done this before like yet like everyone has this need and yet like like it's taken into like dr eb to be like hey we're gonna create like something that's very like sequel we're not gonna like throw it out and try and create something completely new but let's just make a few little tweaks here and there to make it like a little bit more logical a little bit more useful. Yeah and i think that's the number one weirdest thing of sql as a language is like it's not one thing but there's like a standard that everyone adheres to until um they don't and like i i feel like that's like i've loved most of the changes they've dropped and of course introducing things like column selectors which i'm a very big fan of largely thanks to dplyr dplyr but at the same time like that is it's a tough trade-off too because i know everything like every time you deviate from the grain it's like yes but it's also like slightly less interoperable.

Sequel is really interesting to me because it's sort of like this pre-internet technology in many ways like you know sequel is like a spec it's like this like an anti-sequel spec but you cannot actually get the spec without like paying for it so i actually have like one of the few like books i still like i have sequel 99 like complete really like this this is the only resource i've been able to find that actually like explains how like sql is supposed to work not how like some specific database implements it and i think that's i don't know i find that kind of bizarre i mean luckily like i think recently like claude is like filling a lot of that gap for me like claude seems to have this knowledge of sql that i could never like through googling i could never find the right website but it must be on the internet because like claude seems to understand it now um but i don't know it's just really interesting to me as someone who's been like writing this translation layer from r to sql like to try and figure out like like what like what is sql like what's the official way you're supposed to do this and like what the databases actually support and like what are the kind of databases that are like you know kind of growing up you just just you sort of assume whatever database you use the most of is like the normal database and everything else is weird but like getting out of that mindset being like oh okay this is what it's supposed to be like and every every database actually does this incorrectly.

That is fascinating about the spec i didn't realize that was so not accessible like there's so many like fascinating topics in open source governance but i've never heard of that like truth rules but you can't see them i mean you can get them but it costs like three thousand dollars or something and i mean maybe it's easier maybe i just never had the right searches but that just yeah i found that like mystifying like and kind of even just this idea that like to me like if i want people to like do something to like follow something i've written like to me it's like obvious you want to give that away for free like oh yeah you don't need to charge people part of the like bit of success you talk about too of just yeah get your stuff and then yeah and then i think you know some of the most successful database like oracle and sql server they're the ones that are most distant from the from the spec.

Yeah it's interesting too the spec because it's so expensive like when i was working on sql translation question i had was like if i bought this spec would it even would it just be a prescription or would it describe databases as they exist like it was almost hard for me to tell like did databases follow this or is it kind of a prescription that got i think if you read it it's like also i i think like if you were to read it it'd be actually extremely difficult to understand because it goes into a very very high level of detail and you actually want like something kind of in between you want like the general like what is this supposed to do but not every single possible little edge case like if you've ever read like the html5 specification or anything's like really to get like then to just lock down every edge case so people can agree on like this is the way to do it it's so much detail i mean i think there was an aspiration to make it like to make it actually a standard like in the sense that c++ has a standard and then compilers agree to to implement that standard so you can say a c++ 14 c++ 17 c++ 20 and so i think sql they tried to do that but then the reality was that you know the business needs of database vendors trumped uh conforming to the standard and so they would introduce things that went outside the standard or of course things like function names weren't standardized and so every database has got different different names for functions and like some some databases implement division differently like whether you can divide by zero like division by zero has like a prescribed behavior in the standard but then some some databases will.

Yeah mind-blowing thing to me is like sql doesn't like in general databases don't support proper floating point math like they don't follow you know every you know the way you expect like modular arithmetic and division and stuff to work and like r and python in every modern language like sql is different like you can't represent like sql in general doesn't have like a notion of positive and negative infinity or not a number it's just like sort of fascinating that this like technology like this dinosaur technology still like still still lives today.

Column selectors

Yeah I think just to like circle back and flesh out something you brought up emily is you mentioned selectors and i wonder if it almost be useful to almost explain what a selector is um because it's kind of like it has been added recently to a lot of sql implementations but maybe you could talk us through like just what is a selector. Absolutely and i think that tees me up for a question i've always wanted to ask cadley so um appreciate that but um so if you think about i mean i'll pander to the art crowd here to start out but as selectors i mean you have a data frame you have a bunch of columns um if you have in fact named your columns well in a standardized way or if you think about data types there are a lot of different like kind of identifying information you could use to grab out and act on a set of variables um so if you want to do some sort of mass data angling process like say in sql maybe you want to take the average of every boolean variable you have in your table if you're doing that in sql you're going to be there a while because you're going to be typing like mean or average variable a average variable two average variable three um but in some of the more modern um kind of programming languages with more flexible apis you can have these like really nice kind of selectors where you might say like for all boolean values apply the same transformation or um if you get a little clever with like naming columns in a standardized way for all of my like variables related to this entity you're representing an indicator do this operation um so for kind of like large scalar angling it can just be a way to like write a lot like cleaner code avoid a lot of like kind of typos or copy paste errors by consolidating your business logic and just applying or mapping it over many different variables.

business logic and just applying or mapping it over many different variables.

And i know the first place i saw this was in dplyr i think now i know um it's in spark has it pullers has it a number of different like sql or sql variants have it pandas has it but um heavily like i've always wondered like to the best of your knowledge was dplyr the first tool that had implemented that sort of those are those selectors. I don't i don't know it seems like my recollection of this is vague like i kind of remember around this time like kind of learning that oh like one of the problems people have is that they have a data frame with like 800 columns and just selecting the correct columns is a pain i my vague recollection and i did a little googling that kind of supports this is i think like stata had some tools for variable selection uh that i that i have i and i think like some i think that's where i learned about it from from like stata users um like looking at the documentation for stata like it looks like it you can say like here's a here's a start of the range it has their end of the range here's like all the very you could select all the variables of the prefix so i i think that's probably i have a very vague relation recollection that maybe sass has something similar as well but yeah i think i think that's or i think it came from like statistical software which is kind of surprising because it does feel like this is something you feel the need for and like sql all the time because you're just typing out like i was just looking at looking at pandas to see if it has it had it and i don't i don't think it does i mean i haven't been super active in pandas in a long time but um it doesn't it doesn't look like it has selectors quite in the same way that that dplyr does or that you know duck db does now for example i think the ibis team implemented it um and i guess like polars and polars and ibis have selectors similar to dplyr now. I knew i'd done something similar in polars or in python with pandas of just grabbing out the columns and doing some list comprehensions and throwing that back in but i didn't want to say that was the best way to go.

but i'm fascinated if it came from stata like my main recollection of my major collection of stata from college is like you can't have two data frames in memory at the same time so i did not think of these bits as like bastions of user experience

so interestingly like i'm looking at the docs for this on the stata web page and it one of the things that mentions like there's specifically like my kind of favorite selector in dplyr that like literally no one uses is called num range and so you give it like a prefix you give it like a starting number and an ending number and it will generate so you can say like select me from x1 to x50 really easily and like the stata docs specifically say that there's no way to do that easily in stata so i i don't know i think that's kind of evidence for like okay i was like well stata can't do this like i think that's something useful and i'm gonna do that

you mean there's like that's like spite like a response evidence of a response you're like

yeah but it's also one of those functions i'm like i think this is cool i would have thought like people would use this and like basically no one does yeah it's so fun it's such a small in a way like such a small behavior but it does seem like so like it's it's so ubiquitous that that it does make a big difference that that sql users and when i've had to type sql manually this idea that you just type over and over again the same calculation say on and just change out column names um is kind of is kind of mind-blowing so it is funny like selection does seem like a really simple topic but it's kind of crazy how much it shapes the quality of life and i think even like for me personally i always wish like i feel like if that existed in databases natively people would think about more like as they build their database is like names is something that can do a thing and then think about those more cautious carefully like you would think about designing an api i think you know when you get into like kind of industry or production databases it's like you get into all these things where there are like 10 different ways to like abbreviate account id so you'll end up with like 10 different versions of that be mentally accounting which table

but i think for me it was just like a big aha moment of like oh like my column names can actually like do something if i'd like actually think of them as like part of the software and this i think this is very much your um column names as contracts post is that right yeah yeah indeed indeed yeah

but i think for me it was just like a big aha moment of like oh like my column names can actually like do something if i'd like actually think of them as like part of the software

nice and you i feel like dbt plier so like your dbt plugin kind of like built on that concept a bit do you try to explain a little bit about that yeah exactly so um dbt um plugins packages whatever you want to call them are the answer to what kind of was called out about um sql not really having a native function interface again i think that's something that's wildly data specific some do and some don't but even the ones that do it's not really great to use them because then you've just like loaded code to a database it's not really version controlled you can't really see what it does can't access it that easily but um dbt just i guess taking a step back is essentially like a collection of sql scripts macros and other files organized in a very specific way so it can kind of infer the dependency graph and execute um sort of similar to how in our packages maybe just a lot of our files organized in a smart way so the computer knows what to do with it so they abstracted that then like step forward more where um you can have dbt packages which are kind of like data agnostic chunks of sql code that you can then like kind of import in calls macros they can be at the function level they can be at the table level they can exist in a lot of kind of different levels of granularity um so dbt plier was kind of like an early-ish um dbt package that i put out there that was essentially stealing um a number of things i really liked about the tidyverse but specifically around column selectors and trying to like port that api um into sql to solve this exact kind of problem of how do i grab out a set of columns and how can i then like apply transformations on those in bulk

Reception at dbt's Coalesce conference

what was the reception like because when i i went to dbt's conference coalesce in 2022 to give a talk and they were like they were kind of surprised that a person from RStudio came through they were excited but they the one thing i remember is they kept saying like emily is our like representative from the art community and they kept like they they pointed to dbt player a lot as an example of kind of like a really interesting cross-pollination of ideas like what was the what was the reception like what was it like kind of going into that community i think it was interesting because i think there were people kind of going back to our the point about it being like a simple idea but that has legs of like there are people that like got it probably that had seen it before i know there is one um dbt labs engineer at the time that um really was like knew a lot about our had been part of that community really kind of like like latched on to it you know i mean i think people that had like seen how it worked somewhere else um kind of really got it

i think the thing i realized in retrospect was probably calling it like dbt selectors um would have probably been like far more useful and informative for like discoverability purposes um since like dbt players like kind of a shibboleth of like pre-limiting yourself to only having like the art part of the community having like any earthly idea of what you're talking about or what to expect from it yeah that's a good point that's a i feel like a perennial uh forever debate the like should i name it after the thing it like most resembles for me or should i give it like a new name for these people like over here yeah and like in my head it was like an homage kind of to like db player which i'd also like used a lot got a ton of value from um but naming is hard

Naming packages across languages

that makes me think of a question we've been struggling with that you you might have some insight on and that is like as we're creating more uh packages where we have like an r version and a python version like what do we name them like do we give them like i think with like orbital and pins we're like okay we're going to call the r package and the python package exactly the same thing or do we do it like um great tables we've got gt on one hand and great tables on the other and i think there's another i don't know if you remember michael but i think there's another case where we're like these are totally different names like do you do you have any sense of like like what do you think we should do when we're doing like one package like the same idea but implemented in in two places in two languages

that's a really interesting question because like i'm sure too you're also bound by different namespace availability in both languages yes on one hand like like with orbital like i find it very satisfying of like if i've heard about in one language then it's trivial easy for me to be like oh yeah i know they have that thing in the other language but i do think i mean even with like gt versus great tables like in a weird way there's like a nice like mental name spacing to the fact that like i know these do not like aspire to be at parity the apis within them aren't 100 the same and like i feel like it kind of fits my expectations right that like these are aiming at the same goal but they may not 100 get there in the exact same way

yeah that's interesting because like orbital is the case we probably that like the api of orbital is simple enough you can base it's basically the same for r and python but obviously the more complicated the package the more it has to kind of diverge like we like you want a package that feels r-like and you want a package that feels python-like you don't want to be like oh i'm writing r code in python or i'm writing python code in r that's that's something i've so loved about um michael and rich's work on both like kind of great tables and um point blank and i mean honestly even like plot night i feel like manages to get like that distinctly more like pythonic feel in the python version and i do think that like something i think a lot a lot about is i like kind of switch between languages is like i'm all very like i definitely write python code with an r accent but at the same time like as best like it feels so much like you're going to get so much further and have a lot less pain if you are like leaning into the conventions of the language you're in

i definitely write python code with an r accent but at the same time like as best like it feels so much like you're going to get so much further and have a lot less pain if you are like leaning into the conventions of the language you're in

so i've really it's been fun from the outside looking and i feel like to see like where the scientists i'd same and where they do kind of like for goth i think i think ours had ours had a lot of impact on on on python as well and and if you look at like the api of api of polars api of ibis and it's it's a much more like pipe centric kind of fluent you know fluent api and and i think um that's you know people looked i think a lot of people myself included looked enviously from the python world into the r world and then we were like oh wouldn't it be nice to not to just be able to refer to column names like inside of expressions and not need to like you know sub basically you know index into a data frame to be able to get a reference to that column to be able to do things and stuff like that like the non-standard evaluation that you have an r that comes from its its lisp uh lisp heritage and so i think that the people in python have tried to replicate like some of the some of the flavor or like the the the niceties of that like the underscore like the underscore operator and and stuff like that but still it's it's hard to shoehorn that type of api into into python which is like in many ways the opposite of non-standard evaluation like it's the you know it's everything must be as explicit and obvious as possible and so to try to do things like magically is is considered unpythonic and so that there's always been a little bit of attention there like how do we you know do nice things on behalf of the user but like still make it obvious what's going on at any given time

yeah and that that feels like the kind of like it's almost like the spectrum of like do we give these things the same name is like the very first question and then like that that reminds me of like the deeper questions of like how do we even architect it so it feels pythonic versus like uh yeah it has like an r sort of smell to it how do we actually lean into kind of python as a language it seems yeah it seems so tricky to kind of strike that balance kind of on all those fronts i will say for great tables we when we started working on great tables which is a port of the library gt in art of python it wasn't available on pypi so that kicked off right away like the need for a new name but what i appreciate about rich is he was open to like what if we just nobody few people knew that gt stood for grammar of tables so i have to hand it to him for being willing to just retroactively like rewrite history and pretend the acronym stood for great tables i think i think actually few developers would be willing to retroactively change and i'd actually forgotten that like in my head i'd already translated yeah like gt that stands for great tables i'd forgotten it was like grammar of tables inspired yeah that's like perfect i didn't know that story so that's that's that's cool

yeah kind of freaky but i do think on the name front it's challenging because i think i i'm biased but i think it's hard like when you give something the same i think when you give something a new name it's a chance to really like speak exact just to that audience um and like really force yourself to address them kind of directly i mean i do i do think one of the problems with the python community is that they don't appreciate puns as much as the community so i don't know like i think there's a lot of like r package names that like just so perfect for r and just like would not be would not feel right in python i mean the other the other big problem with the python community is like not enough hicks or not enough logos like people don't do logos for their packages i'm like what what is this this is a serious problem this is like this is not a trivial problem because i found like if you go to like try to give a talk about python you're like i don't know what to put on the slide you know i mean like r it's like you need a simple slide you get peck sticker

but it's a it's a struggle yeah i mean that was great tables i think we used a pentagon specifically to have a logo but for it not to be a hex which i know had the uh maybe objects to aesthetically but um i don't know i think there's like something perfect about like the the python package logo like not tessellating the plane it just feels like python like we don't want to yeah exactly exactly like ah we're like let's all fit together let's all the packages work together

How Emily chooses what to explore

yeah i'm curious emily how do you choose like the things you get into because i think that like digging into column names as contracts and like addressing the dbt communities so interesting um how do you kind of choose what what you're gonna like which threads you want to pull i think i've always like first just like really gravitated towards things that might otherwise frustrate me but just like had so much curiosity of like couldn't they be better like don't we maybe need more internal packages or like surely there must be a better way to like write the sql code and i think so i think kind of like really being kind of like drawn to kind of like the little just like more paper cut everyday problems um i think it's just like left me with a lot of like curiosity and energy to explore things that otherwise it just kind of like train you and then i think that's just coupled with like i love just like learning about kind of new new tools new algorithms you know just like kind of squirreling away information that like it doesn't feel like i probably ever need to have just like save it as a nut for winter and um i think and i think that was so easy especially

early in my career was kind of like the renaissance of like our stats twitter where you just have like people just like you could be on the fly on the wall for just like non-stop conversations people so much smarter than you doing just fascinating things and then just like very luckily just being able to do a lot of that pattern matching of like oh i have this problem oh and then i heard about dbt or like i'd really love to never copy paste this again but i think there's a thing called that aren't markdown that exists um that i think is just like it's kind of like i'm curious about the things that frustrate me and then i had to have all these like tools i want to try and then sometimes i just like kind of get lucky pulling the right tool from the toolbox

Learning in public and imposter syndrome

yeah it's so cool and i i do think it's so interesting where so much of this really shines in your blog where you're often like stitching together tools like multiple tools in really creative ways and i know i think something that that like came up when when preparing for this is like imposter syndrome because i know that like with blogs or putting when people put themselves out there like that's a big can be a really big challenge i'm really curious like your take on like what what that looks like for you and maybe advice you'd have for people that are like blogging or putting tools out that might feel a little imposter syndrome

absolutely i mean like even preparing to join this podcast is like what have i done i could just be having a nice afternoon and i'm talking to like three luminaries this field but um i mean i think in some ways i think it's something that in some ways you really have to get comfortable with because if you like if you aren't in rooms with people that you feel like you have a ton to learn from that's kind of sad you're probably also in the wrong rooms you know i mean going back to like i think the last question of like the way i learned was just getting to like absorb all this amazing content from other people out there and like if you weren't part of the conversation you weren't going to get to kind of like learn and encounter those things so i think like i've always like somehow managed like get myself like in a lot of places that i really had no business being you know even switching between like the analyst engineering more mo hats and like just getting kind of comfortable jumping in and like learning as you go and same with like obviously not phd political scientists or economists but the more that like i could like peer over into those spaces and see the different ways they're talking about causal inference you see you know i mean like i think to a large extent i think it's very good to feel like you aren't smart because that's like tells you you're like in the right place where you're learning

sharing on the internet i think it's always like hard and scary i think it is so much to the great credit to the art community that like it never felt that way i've thought about that if i graduated like if i'd like come into the field either like five years sooner or five years later both of which rudely was when the internet seems like was most more hostile place and like oh what i have felt is like confident just like putting stuff out there but um i think emily robinson really maybe said it best at some point in a um conference talk of like just like right for the person that like is struggling with the things you were struggling like 6 or 12 months ago and just like try to help them know what you need to know because i think that's just like a good way to remember there are other people out there just working on the same stuff then that you do like genuinely have something something to contribute

just like right for the person that like is struggling with the things you were struggling like 6 or 12 months ago and just like try to help them know what you need to know because i think that's just like a good way to remember there are other people out there just working on the same stuff then that you do like genuinely have something something to contribute

yeah it's such a neat um way to frame it i feel like of just like thinking back as both a way to like really emphasize like your own growth but also like it does it does feel nice to be able to just take that difference and like write it down yeah and and i just like i think it's the best way to like crystallize your own thoughts that i mean nothing i write will ever be like hey guys this is exactly what i did at work today because i'm pretty sure my employer would um they kind of prefer i not go like sharing ip all over the internet but um i think kind of also that just thinking about like how would i help somebody else do this thing forces you to actually think at the right level of abstraction that i mean to just take the trivial example of column names it's not like how it actually works better for the state is that if i do this but kind of like almost like it forces you to kind of like generate like a theory behind things that is in itself a little bit more like reusable

What we'd tell ourselves 6–12 months ago

i'm curious if folks have this is maybe a tough exercise but like things they would say to themselves from 6 to 10 months ago 6 to 12 months ago like i'm almost curious to hear what people would tell themselves i mean i think i would tell myself to like try claude code earlier maybe i was trying to use that like six months ago but uh i think that is something it does feel that like as you get further along in your career though like you do like the your growth definitely slows like there is that period like when you're first starting your job and there's just like this firehose of information and everything is like oh my god this is like amazing and you look back at code you wrote like six months ago and you're like this is a heap of shit like why would i ever write that but certainly feels like as you're kind of you do plateau as you get further on your career i do i guess that is more my worry now that like i am plateauing like when i look back at my code from a year ago i'm like oh that's pretty good like i don't know that feels kind of bad like i should be like oh that's kind of shitty headley you could do way better now

but uh i think for me like you know i i started using code like coding agents basically as soon as i learned about claude code about i i think it was in march i think maybe it was was it released in february like it wasn't it was definitely like beginning of this year so there was a little bit of it took a little time for me to find out about it and then start using it but i i think i for a long time i i um i held off for no particular good particularly good reason on building like bespoke tools for myself like essentially identifying things that like in the in the past like five years ago you know you know those little i think there's even an xkcd comic about it it's like the grid of like is it worth building a coding a solution for a problem how much time per day will it save you and that tells you whether it's worth it and so now with coding agents like that whole chart needs to get completely redone because the amount of time that it takes especially if it's building something it's not very hard to build but it's something that is just for you and it makes your life just a little bit better it saves you five minutes a day 10 minutes a day maybe an hour now and then um i i you know should have been a little bit more like i i've built things in the last three months where i'm like should have done this a year six months ago or or or a year ago and so i think that that seeing those those successes have given has given me like a sense of like you know maybe more boldness or like more of like a willingness to like dive into things or like set you know set an agent to work building building something like maybe it's only something it's going to save me 10 minutes like twice a month but whenever i have those 10 minutes that it saves me it's going to be super satisfying um so that's definitely changed my whole my whole perspective on like how i do work now so it's it's uh um you know i think there's no way to really internalize that that knowledge without going through the experience but so i don't know if somebody like hit me over the head with that like west like build a bunch of custom tools for yourself to automate your your life and your development work um if i would have believed them that it was possible nine months ago but uh i certainly believe it now

yeah i mean i i hate to add to the claude code but i think i'm similar like i think it was in a podcast with james blair where i realized i was the only one not using claude code which was a good realization but um now i'm using claude code a lot but i think that the most interesting thing and surprising thing to me has been like when i watch my friends use claude code i'm always i always learn new things and so i feel like both i would tell myself to use claude code sooner but also like watch watch people use it because i feel like actually with claude code it's so easy to just go on your own and do stuff but i'm endlessly surprised at how differently people approach things

that reminds me a lot of this thing i remember like like like if you're a data scientist working with like a you know someone non-technical often the things they think are going to be really hard are often really easy and the things they think are going to be easier often really hard and we're kind of in the same state with like claude code and these tools we don't actually know like what is hard and easy for them and it's so easy to like get stuck in your like you have to see other people do it and you're like oh i just assumed claude code could never do that it'd be too hard for it and it actually turns out to be like an easy problem for it for whatever reasons it's really like it's like fascinating learning this whole new like way of working with this like alien intelligence

Tacit knowledge and learning from others

i feel like that abstracts to so many things just like the ability to like learn from watching other people do things which i mean is such a like hallmark too of like think r and python whereas like i think that's something i've thought about a lot i mean also going back to just like sql of like there's so much less open source sql code on the internet because it's mostly probably like code people wrote for um business applications and like i'm always like fascinated by these very little spaces where it's like we are like less able to like learn from one another

it is yeah it's also just hard to like share like this is one of the things i remember when like trying to teach sql like to teach it you've got to like spin up a database like yeah it's like really how like how do you simulate that environment and that like that was definitely one of the things that always like motivated me and the design of our packages like how do you get people like to experience that kind of time from like downloading it to experiencing a win and like if you have to go away and like install postgres on your computer then that's like a multi-day journey that involves a lot of pain and suffering and by the time you're like by the time you've got that installed you're not going to like follow through some blog posts about learning and the way i mean i feel like with our teaching you always like drop people like directly into the analysis like i remember the first time i read r for data science i was like oh this is cool because this book isn't trying to teach me what a for loop is for the amount of time it's fine i get it you're just more like hey like here's the diamond status set let's like jump between some like wrangling and some biz you know this makes a good question and like that too is just i feel like it was kind of the analogous of like watching someone over the shoulder do it um which i think can be like a really struggle earlier in your career if like you've never actually gotten to see like what good looks like like how can you aspire to it

one of the um words i thought the phrases i found super useful was this idea of tacit knowledge which i learned from bill bearman who taught this like fantastic amazing data science course at stanford that i can help out with and this idea of all of these things that like you know to do but you'd never think to write down and like when you watch someone working you're like oh my god you can do that like like it's fascinating like even you know within the tidyverse team we've like worked very closely with each other for a long time like still when someone shares their screen and they do something and you're like like what you i was like what you can right click on the positron in the dark and it lists your like recent projects like like mind blown like you'd never think to write that down because it's just something you're like oh everyone knows this and like seeing people like in the wild like do analysis like do analysis is just is really cool

oh yeah and i think that almost gets us back to the imposter syndrome too of just like anything anybody can put out in the world is that like net new aha moment for someone like i've always felt like to the extent that i have a niche it's just like writing about things that were like too boring or ordinary or table stakes that no one else ever bothered to like think they'd be interesting to put pen on paper on but um it is just like really satisfying when you just like find that tiny little thing that was too unimportant to say that actually just feels so good

like to the extent that i have a niche it's just like writing about things that were like too boring or ordinary or table stakes that no one else ever bothered to like think they'd be interesting to put pen on paper on but um it is just like really satisfying when you just like find that tiny little thing that was too unimportant to say that actually just feels so good

yeah i mean i think that like ties kind of perfectly back to like the column selectors which just feel like this like trivial little thing but until you know all people have thousands of columns in their databases and this is actually a really painful problem and like let's just create like one like boring little helper that just unlocks you know gets rid of all this pain oh yeah yeah i do feel like selectors too it's the thing you appreciate when you watch someone coding and typing you're like this is so fast actually like you kind of have to see people do it to really get just how powerful it is

Wrapping up and posit::conf(2026)

but i think we they're they're kicking us out you know we're getting long in the tooth here uh i emily i love i appreciate so much you coming on i feel like it's been so neat to talk about all these things and both like how you've spanned communities and but also like i i appreciate you surfacing the origins of selectors through stata um and this like really important advice for people with imposter syndrome and how to just get into different areas and really like embrace discomfort and like talk yeah address yourself from 6 to 12 months ago

i'm also i had the are we allowed to talk about posit conf yeah sure sure if emily's okay with it i heard a little bird told me you might be the keynote at the next posit conf is that yeah i am both honored and i told um jenny when she mailed me i'm right now i'm still speechless which is probably not a good trade in keynote speaker but no i'm i'm so excited yeah we're super excited to have you well thank you i'm honored going back to imposter syndrome a little bit horrified but um honored yeah looking looking forward to the journey because i feel

like i always like i learned so much in the process of like bubbling my thoughts back down down into words like i feel like the the process of writing talks too i feel like you just learned so much more about whatever it is you thought you were enough of an expert to talk about yeah 100 well i feel like we're so lucky to have you at posit conf and i'm so excited for the talk and really appreciate you coming on no truly thank you all and thanks for the great show i'm looking forward to being back in back as a listener next week yeah awesome thanks

the test set is a production of posit pbc an open source and enterprise tooling data science software company this episode was produced in collaboration with creative studio agi for more episodes visit the test set.co or find us on your favorite podcast platform