image: thumbnail.jpg
Rebecca Barter: Persistent learning, tool building, and ‘Will code even exist?’
She shares how moving between R, Python, SQL, and dashboards reshaped how she thinks about expertise. Plus a reflection on her recent posit::conf talk, “AI: Hype, Help, or Hindrance.”
Rebecca digs into what it’s really like to work with AI every day and why humans still rule, especially in exploratory data analysis. She explains how tool building can be the fastest way out of busywork and how teaching beginners sharpened her ability to communicate clearly. The conversation circles a bigger question too: As AI keeps improving, are we headed toward a future where code looks completely different … or maybe disappears altogether?
IN THIS EPISODE
• Why motivation matters even more than productivity
• Escaping busywork by building better tools
• From R to Python to dashboards: Learning fast as a survival skill
• Reality check on AI in the IDE
• Why exploratory analysis still needs human intuition
• The 80/20 of coding: Automate the boring, protect the judgment
• Teaching beginners and earning trust
• The uncertain future of code
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning, digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
On this episode, we sit down with Rebecca Barter, Senior Data Scientist at ARENE and Adjunct Assistant Professor at University of Utah, who has really interesting perspectives on learning and teaching and demystifying AI.
Hey, everyone. Welcome to The Test Set, where we dig into the people behind the data. I'm joined here with Rebecca Barter, who's a tenure data scientist at ARENE and an adjunct professor at University of Utah. And I'm joined by my co-hosts, Wes McKinney, who's a Principal Architect at Posit, and Hadley Wickham, who's Chief Scientist at Posit. Rebecca, we're so happy to have you.
You know, I thought as maybe a quick icebreaker, I could just start off with something you said. There's so much I want to get into, but just really quick. You mentioned a lot in prepping about AI and teaching and kind of your path. But one thing that stood out to me was you said, you're not the kind of person who can work on something you don't care about. And I feel like that is so relatable. I thought maybe you could just kick us off with a little bit of that.
Absolutely. Yeah, no, I get bored easily, I think is really what happens. So if I'm given a bunch of tasks or like working on a kind of field that I'm like, there's no point to doing any of this. It's just like mindless tasks. I start just staring out the window thinking about what I'm doing with my life. So I'm very much like, if I wake up in the morning and I have tasks where I'm like, yeah, I actually care about completing these, I feel like there's some kind of impact, whether it's to other people, to the company that I work for, or to a field as a whole, I'm going to be so much more engaged. I'm going to actively have fun. Whereas if I wake up and I'm like, all right, let's just kind of get butt in seat and do the thing I got to do. And it doesn't really matter if I do it or not, but I have to do it. I'm going to probably start crying at some point. That's just, it's not me. Some people can.
No, that's so real. And you're, as I understand it, you're in Tahoe, so you live in the mountains. So if you're not motivated, also you have all these beautiful mountains.
Exactly. And I've definitely done that. So I was, before I was living in Tahoe, I was living in Salt Lake City and it was the same thing. So there was like mountains right there. So like, you know, anytime I start feeling that feeling of like, I don't want to be doing this, I find some kind of excuse to just like go run away in the mountains and stop doing the task.
No, I feel that so hard. I'm just, yeah. Wes, how do you feel about, can you grind through things you don't want to do?
So sometimes you have to do work that you'd rather not, but it's for the greater good or like the project just is not going to be able to move forward without it. So I've definitely found myself like dealing with, like thinking back on the early days of Pandas, it was definitely everything having to do with Windows was like the thing that I didn't want to do. And so there was a dark period where I had like my Windows virtual machine was like the blessed place where I would create the installer packages for Pandas.
This was before we had wheels. This was before, you know, Conda and UV, it was like dark times creating these .exe files to install Pandas. And at no point did that ever spark joy. This was purely, you know, work that needed to get done for the greater good of the project.
Yeah, that's, I could see Windows, no shade on Windows, but yeah, sometimes Windows is being the demotivator.
JJ used to say that, like, I think it was some company, he was like saying, like, if you want people to work on Windows, particularly Windows installers, like those people just need like an automatic bonus because it's just like such, it's like such important work, but like no one wants to do it. And it's painful.
I think that that comes around in a lot of aspects as well. I feel like whenever I'm teaching a workshop, it's always the people on Windows who have like installation issues. And like, I'm like, I have no idea how to help you because I don't even know how to use a Windows machine like normally.
Work at Arine and tool building
Yeah, that's real. Well, I hope that you're working on a lot of things you do want to do right now. And I think I'm, yeah, I'm so curious to dig into a bit about your like AI work and what you're doing as a data scientist now. I figured what one approach that might be nice is maybe you could tell us a little bit about your work at Areen now. And then I'd love to circle back and maybe talk about kind of your path to getting there a little bit.
Absolutely. Yeah. So, I mean, I think related to your previous question as well of like, you know, I like to do things that I'm passionate about really is kind of what it's about is the thing that's really got me going in terms of data science has been like applications in healthcare and like the kind of patient focused approach to analysis and trying to like help improve patient experiences. And so that's really a big part of what's driven me to my current position, which is at Areen, which is a kind of medication management startup where we're really trying to like, make sure people are taking like correct medications and they're, you know, not, they're adhering to their medications and like all of stuff like that, where it's just like really kind of focusing on the patient experience and trying to make sure like we're improving health outcomes. So like that for me is something that I can wake up in the morning and be like, yeah, stoked to do that, you know?
So yeah, I think it's like, like for, for me, I'm very much drawn to, to these kinds of healthcare applications in general, but it's also like, I'm lucky that I get to work with a lot of really wonderful, intelligent people who are kind and, you know, like just the joy to, to, to work with as well, which makes a big difference.
Yeah, that's awesome. What's the team you're on, Mike? Like, what's the, like, how many people do you work with and what's the setup like?
Yeah, we're a team of like, I think maybe 10 to 12 people. We're kind of split between like health outcomes and data science. And I'm, I kind of do both. I do a lot of like tool building, like, so, you know, if there's a lot of kind of tech debt and like things that have been built that are really kind of ad hoc and inefficient, I do a lot of like, kind of working with whoever wrote that original code to like, be like, let's create this, just turn this into a pipeline where it's like, you just run a script. It maybe does something with like Quarto even down the end and like creates this nice report. And like, you don't have to like, like, like, you know, run this and then run this and then change this config and then change that config and then run that and then run that. And then, oh, you want something else? Okay, let's do it all again. You know, like, like, like I, like the people that I work with are a big combination of like engineers, like people with stats background, people who are more data kind of science background. Like it's a pretty diverse team. And it's, it's really cool. Like everyone's really easy to work with and really friendly.
Yeah. Oh, cool. Yeah. I love the, like the tooler role where you get to kind of roll in and help like spruce things up and like put some glitter on it. How is it, how do you find, like, how do you choose what to beef up or like which tools, like who needs or what needs a little bit of like tool love?
Yeah. Usually there's a, like, it's pretty clear, like when we've got inefficient pipelines, you know, like, I think it's also, you know, a nature of, of being a startup where a lot of things are built kind of ad hoc and, and for individual customers. And then it's like, oh, we need to kind of scale this up to be more general and more applicable. So, you know, it's a lot of the time, it's like, there's a lot of things that we do repeatedly for customers that like, we need, it's like very clear. It's like, this could be improved.
So like, like it's pretty easy to point towards, okay, that particular pipeline is really inefficient and could very much be like benefited from having someone create kind of a overall compute pipeline that makes it a lot easier to go from point A to point B, you know, rather than going point A, A.1, A.2, A.3, and then eventually you get to point B and you realize a lot of things that you could have done differently along the way.
Picking up new tools and languages
Yeah, yeah, for sure. And I, I think you mentioned before this that you, I think you, I've seen you've done a lot of R, work in R, but that you might be doing a little more like Python and stuff now. I'm curious the kind of mix of tools you're using in your work now.
Yeah. So for a very long time, like I learned R when I was like 18, you know, I'm not going to tell you how old I am now, but I'm older than 18. So I learned R before the tidyverse existed, before Hadley had created ggplot, you know, all the good stuff. And that was like very much like, you know, like that was what I was comfortable with. Like, I was like, yeah, I get it. You know, I'm, I'm R, R like, I felt like I was R queen, you know, I'm like, I know how to do R things.
And then Hadley came along and like ruined everything. Because now he's created all these new tools that are way better that I had to go and learn. And I just, I remember someone being like, like, like I was fortunately for me, I was like, I was in grad school by this point. And someone was like, okay, we're going to use this new thing called ggplot and dplyr and this like project or whatever. And I was like, yeah, can't I just do what I've always done? And it was like, no. So that, you know, forced me to learn. And then I realized, like in that experience, I also very much realized like, oh, things do kind of keep changing. You can't kind of just like learn the one thing and then, you know, say, oh, I've mastered it. You know, like it's, you never really master it because what you've mastered today is going to be really irrelevant in like these days, really irrelevant in like three weeks.
You never really master it because what you've mastered today is going to be really irrelevant in like these days, really irrelevant in like three weeks.
But like, you know, back then it was like a couple of years. And so now it's kind of like, so like through, through kind of like grad school, I did a lot of R and like, you know, I feel like I got pretty good at R. I like, you know, mostly like a lot of the stuff that Hadley and co have built, you know, and that was like my day to day toolbox, you know, and like in, in academia, I was in a statistics department, like people are using R that's kind of like the normal thing.
And then kind of gradually as I started trying to like intentionally expose myself to, to projects where I would be forced to learn new tools. I kind of like started, like someone was like, Oh, Hey, I've got this project that like, I realized would end up, you know, I'd have to learn some NLP. I'd have to use Python. And I'm like, the stakes were low, you know? So I was like, this is perfect. Cause like, this will be a really good opportunity for me to like learn like these new tools where it's like, there's no particular like deadlines. Like they don't have any expectations. Like, like they don't know anything about data. So like, it's like a really good opportunity for me to like take the time and be like, okay, that's, I'm going to take on that project. Cause it means I'm going to learn something new.
And this is like, for example, when I was at the University of Utah full time, you know, I found some collaborators in health informatics and health communications that were like, they really wanted help with like a lot of this kind of text data that they had. And so I specifically like, I was like, yeah, I'll help with that and learned a lot of new stuff there. And now I've kind of transitioned to industry. I feel like, and especially with like the advent of AI, like I feel like I've gotten really good at picking up new tools. And I think like, that's one of my main skills is like, like, sure. I'm really good at R and like, I'm decent at Python now, but like, I wouldn't say like, those are the things that are going to propel me in the future. The thing that's really going to propel me in the future is like, you show me a new tool. I'll like play with it and like learn how to use it really quickly. Mostly because like, as long as you can get to like a basic level of competence, like AI can kind of help you get the rest of the way these days, like pretty quickly.
So like in terms of the specific tools that I'm using, like yesterday I used like Python, I used R, I used Quarto, I used Streamlit, which is like Python. It's kind of like shiny, but not really like shiny Python kind of, you know. I stared and got very confused by an AWS step function. Like I'm now, like I use SQL, like every day I'm just like exposed to like such a broad array of tools that some of which I'd never even heard of like six months ago, you know?
Yeah. Yeah. That's cool. You're able to just pick up new tools and kind of move into them faster. And I almost wonder, it's interesting you brought up ggplot2 also as being the one someone might dread learning because I think doesn't, I think ggplot kind of warns people, like I think it gives people a warning, like, hey, it doesn't say you're going to have a bad time, but I think it's like, you're going to need to put some work in before you have like a good time. One of those branding startup messages used to recommend like reading a whole book.
Yeah. I distinctly remember when I first started using ggplot2, I was trying to make just like a simple scatterplot and I just kept getting an error and I did not, I could not figure out what I was doing wrong. I'm sure now thinking back, I was just like missing the AES like function, but like I was getting so frustrated and like, it was just like, like now I look at it and I'm like, well, it's so obvious and it's so clear. And it's just like, I think it's just like when you aren't used to learning new things as well, like, especially like when it's a big change in syntax, it's like, you kind of look at, you look at the code and you can't see what's wrong with it, you know? And it's like, it's also like a different era of like getting help. Like there, like it wasn't obviously where chatbots, like there wasn't even stack overflow. There was like the R help mailing list, which was like pretty, pretty unfriendly.
Yes. I mean, a thing that's a thing that's been keeping me up at night lately,
thinking about is, is, you know, I feel like the whole model for how people discover and start using new open source technologies, like new R libraries, new Python libraries is going to have to change because like our LLM coding agents and assistants are all really good at all of the projects that exist now, which have rich bodies of trading data, training data available on GitHub and stack overflow and all over the internet. But if you build something new, almost by definition, there's not going to be any training data available. And so essentially we're going to have to build things in such a way that we can like point the agents at the projects documentation or like create, create the projects in such a way such that this new thing can be presented up to our, our, you know, AI co-pilots so that they can figure out how to take advantage of something new.
Because otherwise we're going to end up like locked in the present moment of like, nobody uses anything new because if their LLMs don't know how to use it. And so that becomes, you know, the snake has no tail left to eat at a certain point. No, I, I think about that a lot actually. And I encountered that, like, cause like I mentioned, I've been using Quarto, like specifically what I've been doing with Quarto is building like a quite sophisticated Quarto dashboard, like report with like all the tabs and like, you know, cards and all this kind of stuff. And like, they're like, I'm using Gemini cause that's what we use at our company. And like, it does not know anything about Quarto dashboards. And so like my default now, when I'm looking up something and I'm like, oh, how do I do this thing? Is like, I type it into Gemini or GPT or like whatever I'm using. Or like I use the inbuilt kind of like thing in my IDE, like, like that's my go-to now. And it's very clear to me, like when I encounter something that's new, that like, it is kind, it gives you a lot of very confident advice. Like it, it doesn't tell you, Hey, I don't actually know much about this cause it's kind of new, like, or there's not a lot of documentation. And it, it very much like gives you a lot of confident advice and you end up trying to spend a lot of time debugging the terrible code that it gives you because it's just made it up. It doesn't like tell you that it made it up.
And it's very clear to me, like when I encounter something that's new, that like, it is kind, it gives you a lot of very confident advice. Like, or there's not a lot of documentation. And it, it very much like gives you a lot of confident advice and you end up trying to spend a lot of time debugging the terrible code that it gives you because it's just made it up. It doesn't like tell you that it made it up.
And then like, at the same time, like I used to write a lot of blogs, like tutorials and stuff like that on a lot of R stuff and like, you know, coding stuff. And like, there's not nearly as much incentive like to do that anymore as well. Cause it's like, well, people aren't Googling to be like, how do I use the per package? They're going to AI and they're being like, how do I use the per package? And it's going to do a much better job than I ever did. You know? So it's like, things are shifting a lot in terms of how we learn things and how we troubleshoot for sure. And I don't know how that's going to look in terms of new stuff that's coming out that really doesn't have a whole lot of documentation and, and people asking good questions and giving good answers on Stack Overflow and stuff like that. Like, who knows?
Yeah, I do. It's tough too. Like I, I love blogs so much, just like somehow also like encountering a new person with a blog when they have a post that really like resonates with you. So that it is a tough one to like, think like your point about motivation for like certain types of blog posts being lower. I guess maybe one, one thing that would be cool to talk about too is like on the topic of AI, I know you gave the talk AI hype helper hindrance. Maybe you could explain a little bit of that. So that was at PositConf.
The posit::conf AI talk
Yeah, that was a really fun talk to give and to prepare. Actually, it was kind of funny. So I think when I started preparing or like when I proposed the talk, I was very much like, yeah, I know how to use like all these like AI agents and, and whatever ID I was using VS Code a lot. Cause I wanted to use Copilot. That was kind of the main option when I started using it. And then I came around to like the point where I was like, okay, I have to actually like write this talk and like everything had changed. Like literally, like there was like no resemblance between like how I was using it when I started using it to like how I was using it when it actually like, or like how I could have been using it. Cause like, it's another one of those things where it's really easy. You start using it and then you're like, okay, I get it. And then you stop paying attention. You know, you're like, I get it. I don't need to like see all these new things that are happening. But basically like so many, like, you know, things have been added. There was now like an agent, you know, and I kind of focused that particular talk a lot on like the Positron assistant, but it's the same, you know, I'm using like Claude in like PyCharm a lot, like in my current job. And like, you know, VS Code has equivalent stuff as well. Like there's like an agent, there's like an editor, there's like a chat, there's like code completion. There's like all this stuff that is now a part of like these AI tools for coding that like, I really like, wasn't like keeping up. And they're like, I feel like I do this kind of way where it's like, I'll like learn something and then I'll be like, yeah, I get it. And then I'll suddenly realize for some reason, oh wait, it's moved on. And then I'll like have another rapid period of like learning and then I'll like stagnate again. And then I'll be like, oh wait, what? It can do what now? And like, it'll be someone mentioned something and I'm like, what? So like, like that kind of thing is like kind of, it's hard keeping up I think with that. Like I struggle with that for sure.
But I think my conclusion, like really from this talk is like, I've been like using like these tools for quite a while now. I probably started when Copilot was pretty new. And like for the most part, it was like hype, help, or hindrance. Like, well, it's kind of hype. Like it's not really going to do everything for you. Like you have to really be able to tell it what you want to do, what you want it to do. It can't really read between the lines if you give it a very vague kind of prompt. So like, it's not, I don't think it's going to take our jobs, but like, is it like, it's like hindrance, like a little bit, like it gives you wrong code. Like there's like security issues. Like sometimes it just sends you in circles. Like there's a lot of kind of like things that are not great about it, but like mostly it's just really helpful. Like honestly, like all of these tools. So like, this is like Positron Assistant, GitHub Copilot, Cloud Code that are embedded inside of your IDE. Like whether it's Positron or VS Code or PyCharm or a little bit RStudio, that really just has code completion. But you know, like these tools as a whole, like I think I said, like in my talk, it's like for some projects, like they don't really help me much at all. Like if I'm, if it's a pure R project, like I'm so good at R that like, I don't, I don't need, like, I don't have to think that hard, you know? Whereas like, if it's another project where I'm using like Python or like, and you know, writing a bunch of like really complex CTEs in SQL that like, I'm not that good at, you know? Like it means that instead of spending all my time Googling how to like fix the thing that I did or figure out what's the function that I need, it helps me just like, I know exactly what I want to do. Cause like I have that base competence and it just really kind of helps me like go from that, like level of like, I am kind of competent in this, but I'm not an expert to, I can just produce the thing as if I was an expert. Like it really covers that gap really well. But so I think my conclusion was like hype, help, hindrance, all three, but mostly help. And I think, I think that matches pretty closely kind of my experience with them.
Yeah. Was your, did you get any, like in maybe like feedback or response, like did you have a sense for like the audience, how people felt kind of overall? Where were they on the like help, hype, hinder?
Yeah. This is not a continuum. Yeah. I had a lot of people talk to me after, after this talk. And for the most part, it was actually a lot of people who felt like they didn't know how to get started with these things. So they weren't even at the point of, of saying whether it was hype, help, or hindrance. Like they were aware of these things existing, but just like literally didn't know, like how, like, how do you use them? Like what, what are they? You know? So like, I think like it was kind of interesting because preparing for that talk, you know, I was talking to several people during like the speaker coaching where they're like, I think you can assume everyone in the room is going to be familiar with these things. And I was like pushing back a lot being like, I don't think so. Like, like I think there's a lot of people like, just because like, you know, me as someone who's like very enthusiastic about like learning new things, even though it's hard to kind of keep up, like, and like, I think it was a very biased room, like this kind of preparation kind of session where it was like, everyone in that room like is excited about learning new things and is kind of like, you know, whether you're fearful or excited, like you're engaged enough to like go and, and put time into learning these kind of how to use these new tools. But honestly, I think the vast majority of people aren't like that. Like, and that's fine. Like, it's not like a big deal. It's hard. Like I distinctly remember this feeling of like, I'd learned base R. I was like, I'm good. You know, this was the skill that I needed. I needed to learn R. And then it's like, why, why should I learn something new when I can already do everything I need to do just fine? Like, how is it going to help me? And like, it's like, I think the vast, yeah, the vast majority of people who kind of came up and chatted with me afterwards was people just being like, thank you for just showing me how I can use these things. And like, I kind of want to give it a go now. Like, definitely there's a lot of people who like, I think the more experienced you are, the more likely you're going to lean on the hindrance side. And sometimes the newer you are as well, cause like, if you lean on it too heavily, but you don't actually know anything about the language, it's also kind of a hindrance because you can't debug. You don't know how to prompt it well enough to, to be able to make it like do the right thing. So it's, it's kind of a, a spectrum. Like it's not, it's not great for everyone at every stage, I think.
Yeah. I mean, I was going to say like my, my hot take on the matter, like I've been using a cloud code, like Cody agents pretty extensively look like this year. And, but I, I think like so-called divide coding is way, is way over height in the sense like that, that, you know, that, and there's a lot of people like AI boosters going around saying that, that, that soon, soon, soon trademark, there's going to be the coding agents are going to allow somebody without coding skills or data science skills to replace, you know, a senior or expert person in those fields. But you know, as a user of these tools every day, like I, I simply don't see it. Like, I feel like, I feel like being in the loop and reviewing the work that's being produced, it's helping me go faster. It's doing the typing for me. You know, as one person put it, but, but like, I, I feel like if I weren't in the loop, like reviewing the work and giving feedback and catching the mistakes and whatnot, which just sounds like exactly what you're saying that, that yeah, without having somebody having the experience in the loop to be able to judge the output, you could end up, you know, creating a morass that's very, you know, difficult to escape, you know, quite quickly. And so I can imagine in sometime in the next year, we're going to enter some kind of a trough of disillusionment where, you know, a big wave of business users try vibe coding and end up disappointed and conclude that AI sucks. And, you know, was overhyped and oversold. And even though, you know, they feel like they have a gun at their head saying, you know, use AI or else the initial, a lot of the, you know, attempts to eliminate data scientists or eliminate coders are going to fail as a result.
Oh yeah. There's definitely a lot of people who are like not good at using the AI who think it's just like a waste of time and like way overhyped. I don't know. I watched Hadley's keynote at PuzzitConf though, so I'm convinced.
I was interested, I read a good kind of criticism of AI by Corey Doctorow. And I think the thing that he kind of pointed out, like the real kind of risk right now is not that like AI will replace software engineers, which I don't know, like the more I use it and even as the better it gets, I'm like less convinced of that. The real risk though is like CEOs believe that it's going to replace software engineers and hence, you know, start firing them. It's just, yeah, there's clearly like, like I love the tool and how empowering it is, even for me as someone who's like a pretty good programmer, but just like all of the, like the stuff around like concentration of wealth and like the, I don't know, a lot of the people involved and the people who stand to make the most money from this is, it's like pretty, it's pretty icky.
The real risk though is like CEOs believe that it's going to replace software engineers and hence, you know, start firing them.
It is so interesting. I find it really hard sometimes with like a very AI assistant heavy project, like in the early stages too, I find it really hard to like evaluate, is this even going to pay off? Like I just feel like sometimes it gets into states where I'm like, I don't know if I'm just generating a lot of garbage, like how can I kind of like, will I be able to reign this thing in? And I also don't know what the moves are either, even it, it feels like very managerial almost, but also embracing so much production that it's like trying to control like a fire hose sometimes.
But I do think to that point of like, will software engineers be replaced? Do CEOs think software engineers will be replaced? It is such a funny time of like, what is the hard challenges that will kind of like linger? Well, I think for me, I think what's interesting is that, that, you know, coding agents have revealed something that we already knew deep down, which is that probably 80% of the work that we do as programmers is not that special. It's a lot of like configuring, you know, configuring YAML files and writing, you know, grody bash scripts and doing things that are a little bit, you know, essential, but like not, not special. And so, but where the actual, you know, value is in that 20% that, that, that is more special that requires judgment or requires like synthesizing your experience or your background in a certain, in a certain domain, like for example, data science, like requires a lot more human judgment is a lot more subtle than software engineering, which is, you know, software engineering is often implementing something based on a specification and then showing compliance with a, with a specification. Whereas like data science is, can often be as much art as, as science of knowing, like, you know, what are the right tools to employ?
then even interpreting the results may be, you know, may be subjective.
Yep. Rebecca, I'm curious, like at your work, what do you see with AI? Cause it sounds like you're, I mean, you've done a lot of really great work talking about it and kind of helping people like ease into it. Like, what have you seen in terms of people picking it up and trying it out?
Yeah, I think there's big spectrum, right? Like I know people who are like, have no interest in trying it out and using it and incorporating it into their day to day. And like, maybe they don't, if they're only ever doing stuff that's like the same as what they've already done and they're really good at that, like maybe they don't need it, you know, like maybe it doesn't help them that much. But then I also see a lot of people who are like really diving deep into it. You know, like there's a lot of people, you know, at my company and outside, like who, they're like building all these like agents to like do all this stuff, like just to like take as much away from the human as they possibly can, but like in a good way for them, like they're very excited about it. I think other people look at that and they're like, you know, they get that fear. I mean, the first time I ever used it, I definitely had that kind of fear of like, I remember I just asked it, I just said like, write some code to like do, I don't remember what I asked, it was like some kind of R code I asked it to write for me. And it just like did it really well. It was like early days of ChatGPT and I had this like sinking feeling of like, oh no. Yeah, thank you and how dare you.
Yeah, like everything I've like, I got so good at this thing that like I really didn't need to get that good at anymore, you know, I just needed to get like okay at. And like I think a lot of people are, I feel like people, there's so many dimensions and like I've spoken to so many people who are like, some people are really excited about it, some people really hate it, some people think it's stupid and useless and like some people are like very anti, some people are very pro, you know, like it's the entire spectrum, I think. And I don't even know what the distribution is because like I'm probably biased, you know, because like I initially was, I was never anti kind of AI, like it definitely, I think there's a lot of ethical questions and concerns that I have for sure. But like in terms of a tool, like once I started getting better at like using it, like it just increased my efficiency so much that like I now spend, I don't spend that much time doing it, but like, you know, when I'm talking to people about it, I'm usually trying to like convince them how useful it is because a lot of people are skeptical. And like I understand because like I think initially I was kind of like, okay, cool, it can write that code, but like I could also write that code, you know, but it can do so much more, like it, I don't know.
AI for tool building versus data analysis
I know like preparing for this, you mentioned, I think you mentioned like critical thinking skills being a really big piece for a data scientist. I'm curious how that, that kind of plays into this, like the infusion of AI into your work. Yeah, definitely. So it's kind of interesting because I think I use AI a lot more to do tool building than I do for actual analysis.
So like, I think when I'm doing actual like data analysis where I have to like do some explorations and maybe I'll be like, how do I write this complicated SQL thing that I need to write? And like AI will do a better job than me. Like I might use it to like help me like come up with my SQL query. But like, like overall, like other than just like maybe sometimes outsourcing code that I can't immediately like without trying think of how to write, like I don't use AI that much for exploratory stuff. That is still very much just me thinking and me being like, okay, like, okay. I see that there's this many claims. Is that normal? Like how many are there supposed to be? And then I'll go look at like another data set and compare. And like, like, I'll be like, oh, that seems low. Like, you know, and then I'll have to think about like, how do I need to, I need to communicate this to like the right people. How do I want to, based on who it is I'm trying to communicate it to, I need to think about, okay, do I just need to like write a sentence? Do I need to like create a table? Do I need to give them the code? Like, like none of that, AI can't do any of that. And that is like, like, cause my job, I do a lot of tool building, but I also do a lot of analytics. So it's kind of like split 50, 50. The tool building stuff, like AI all day, the analytics stuff, like a little bit of AI, like, like I said, like to kind of help just like speed up the code writing when it's like, it doesn't really matter. I just need the query to be written. And I can't like, it's like, how many CTEs do I need? Like, you know, what's the, how do I lateral flatten? Like, I can't remember any of that stuff. Like, so like for that kind of stuff, it is helpful, but like, like for tool building, I think it's a lot more useful whereas for things where you do need to do a lot more critical thinking and thinking about how to communicate things. Like that's all still, I think me as a human needing to do that.
The tool building stuff, like AI all day, the analytics stuff, like a little bit of AI, like, like I said, like to kind of help just like speed up the code writing but like, like for tool building, I think it's a lot more useful whereas for things where you do need to do a lot more critical thinking and thinking about how to communicate things. Like that's all still, I think me as a human needing to do that.
It's interesting to hear about data analysis that like this activity, when you're in like tool building mode versus like data analysis mode, you're using AI a lot more heavily for the tools. Yeah. I'm curious if you have had like similar experiences with like working on tools versus analyzing data. Like, do you use AI to analyze data ever?
I don't do that much data analysis, but it does seem kind of like fundamentally less useful, at least right now, just because there is that loop. Like you do, like you don't, like you don't know what you're doing. Like sometimes you don't know what you're doing in software engineering, but most of the time you've got like some tasks, you kind of know what you need to do. Whereas a lot of the time in data science, you're just like, well, what, like what's going on here? And there's like a much more of a loop of like, do something, look at the results, do something else, look at the results. And yeah. And bringing in all of your other context about like, like what, what does this variable mean? And like remembering, oh, actually, you know, there was this problem back in 2020 and you can't trust the data before that. And like all the, bringing all of that context, like, like, yes, it would be awesome if that was all written down somewhere. And like, probably we should be striving towards that just so people in our org know, but also so agents in our org can know, but it feels like we're a long way away from that. It's just the kind of tacit knowledge and just like the kind of EDA loop of like, you know, try something, ask a question with a plot, look at the answer, generate another plot and do that as rapidly and many times as you can.
I mean, I've definitely, yeah, I mean, I've definitely used it for like a lot of mundane, like, you know, mundane, mundane data wrangling and, and, and like, you know, one-off simple, like simple queries. Like as an example, like in my home here, I have, I have some like little apps that I've, I've built with cloud code that I run on machines on my, on my home network that I can access over, over tail scale, which is super convenient. But then they have SQLite databases, some of them, and every now and then I'll have a question about like, what's going on with this, something looks wrong in this app. And yes, I could SSH into the machine and I could run, start SQLite three and I could write the SQL query, but, you know, using an agent, I don't have to, and everything is very simple, but rather than something taking me 20 minutes of fiddling, it's something that can be done in, you know, 60 seconds or two minutes. And so that seems like a good, like a good application. But, you know, if I were doing something more, you know, something more advanced or requiring like discretion or judgment, I would be less comfortable. I mean, I think what you said earlier about like, you know, even as software engineers, like 80% of what we do is just not, like, it's not that magical. It's not that, it's the same for data scientists. Like a lot of it's just doing pretty like routine stuff and having something to help you like do that faster is super, super empowering. But that last 20%, like if you completely take the human out of the loop, that starts to get like pretty scary and pretty, pretty dangerous.
Yeah. Like the AI can definitely help kind of take away a lot of those mundane tasks, you know, from us, but then like hopefully give us more time to do a lot more of the stuff that is less mundane, the thinking, you know, whether it's critical or not thinking.
Yeah, that's fair. I find for data analysis, even in the early stages, yeah. If the data's like messy or there's a lot of uncertainty, I think Hadley mentioned this a bit about too, like discovering sometimes the really bizarre, complex things that generated specific kinds of messy data. It is, I find that I have a lot of trouble with like the, almost like the chunk size of the work that an assistant might do with an analysis, like getting spit out like a big blurb or like giant kind of piece of work. Analyzing is pretty overwhelming where like software engineering, I feel like I'm much more accepting of kind of like a lot of work being spit out.
I do feel like it's a bit easier in software engineering too, because you can like double, like you've kind of got this double entry bookkeeping with like code on the one hand and like tests on the other, but like, it doesn't feel like there's an equivalent of that to like for data science. Like you're not writing like reusable functions most of the time. This is like a pure one-off, doesn't make sense for this, this particular data set. And then getting like this big chunk of code, which you're either like trusting or not as a whole seems, yeah, seems more risky. Like a single big pass to a bar chart when you don't know what's real yet, I guess, without the kind of. Yeah, it's on a bunch of like data cleaning and you're like, I'm sure the data needed cleaning, but like, did it make like good decisions along the way? Like that's, yeah, that feels. And then the kind of like, you know, the statistical challenges of like doing this stuff, like you can't, you know, you can't look at the data again and again and again and again in a million different ways. And then, you know, after 17 hours of like torturing the data and it finally confesses to something like, you know, that's no longer like, you can't trust your p-values if you're in some kind of scientific concept and scientific context where that matters. So like, you know, you kind of like, if you're also spending this kind of spending your degrees of freedom, you're spending your ability to, and if it's the model is doing a bad job, you're just kind of biasing the rest of your work. Yeah, it's a tough one.
Teaching and transferable skills
I guess maybe to switch topics a bit. I know, Rebecca, I know you mentioned, you, so you're an adjunct professor also at University of Utah, and you spent a pretty good chunk of time teaching. And I think, is your, you have a PhD in statistics? It's the best PhD to have in statistics.
Wait, no, I was going to say something about how psychology is the best, but I, I take it back. No, I think you're, you're probably right. Do you feel like you probably actually learn more like useful things in psychology classes than you do in like statistics PhD classes? I don't, maybe it's a grass is always greener.
Maybe. It's not too late to bring in a third degree and argue that's the best, you know? Well, I have a, I have a math degree and I dropped out from a statistics PhD, so yeah, I've got, I've got nothing to add. Honestly, I think there's some real power in dropping out. Like actually I would put that as number one. I'm glad I got a PhD, but I do think strategic dropout is another like top contender. Yeah. I kind of weirdly think it's like more brave to drop out than to just finish. Well, the, the, the running, the running joke at the, at the Duke statistics department, whenever I would see my, my advisor, there was Mike West, who basically in time series, you know, specialist. And he always jokes that I was the best student who never graduated. Yeah. It's high praise.
Yeah. I'm just curious about the teaching aspects. So you're, maybe tell us a little bit about your teaching. And I'm so curious to hear kind of like, what, what you learned moving from teaching, from teaching into like data science and industry.
Yeah. I mean, definitely the most transferable skill is being able to explain things clearly. I like that. I can't even say the word clearly when I'm talking about explaining clearly. So, I mean, like, I always really loved teaching. Like when I was in grad school, like, you know, doing my PhD in statistics, I was like, it was very clear to me, like, I am not a statistical researcher. Like that is not a path that is ever going to bring me joy or success. And like, however you want to define that, like, I'm just bad at it. You know, like that was like very obvious to me fairly early on, but I really loved teaching. And so, you know, I did think for a while, maybe I would go down the kind of lecturer path, like be an instructor until I learned about how that is not a particularly viable path to take. Most of the time it's getting better. But I also kind of realized like, like one of the main, like, I love teaching. I think the thing, the thing that I love teaching, love about teaching is explaining things. I love taking someone who doesn't know anything about a topic, like from that point to being like, oh, it's not actually that complicated. Like making it seem, not necessarily like helping them understand a complicated thing, making them realize that it's not that complicated, I think is the thing that I find the most like energizing. And then when I was actually teaching like semester long courses, like teaching a lot of undergrads, like I found that I still loved when I was actually in front of people explaining things like that part, like I really, really enjoy. But then the part where I've got students saying, oh my God, if you don't give me an A, you're ruining my life. Like students saying, I don't want to be here. I have to be here. Like having to like, be like, did you learn this well enough to get this grade? Like having
to like teach, like when you teach in statistics, like if you start teaching upper level classes, you start having to teach a lot of theory that like, I just didn't care about kind of back to like, I can't do things I don't care about. Like little things, sure. But like the big picture, like I just, me teaching a theory class is like, sure, maybe I'd do a decent job at it, but like, I'm not going to have a good time, you know? And so like, I love teaching really intro level things. Like I love really that first step of like, I loved teaching R for example, like teaching R workshops specifically to like faculty, which is what a lot of what I was doing at the University of Utah. I was teaching like, ah, Python, machine learning. Like I was teaching workshops to faculty and postdocs and people in other departments who wanted these skills because they have data and they didn't know how to do anything with it, but they weren't taking it because they wanted to like, they wanted to like get a grade or they needed it on their transcript, or they were just doing it to get a degree. They were taking it because they had a specific reason that they wanted to learn. Right? And so I love teaching in those kinds of environments where it's like very much taking absolute beginners. I love teaching absolute beginners for whatever reason and helping them realize like doing data analysis with R in particular is my, I always love teaching R the most. Like, cause I think like all the tools that Hadley and you know, now everyone for the Tidyverse has created is they're so good for data analysis. Like I think they're so much better than, than like anything else that I've used. And maybe I'm biased just cause that's how I learned it. You know? Like that's what I grew up with in a sense, but like I love taking people from that level of like, they can't do anything. They think it's like so difficult to like, oh, I can actually do some simple analyses. And like, I would teach like a two day workshop. Like I don't necessarily think that's the best format, but like that was what I was doing. And I, a lot of people after that were able to incorporate it into, into what they were doing. And I think really that explanation part of it, that way of communicating that is the thing that I took the most from like my teaching experience into my like job as a data scientist.
Cause it's like so much of what I do is like, I find something in the data or I have a result and I need to explain to people who are not technical. I need to explain to them like, what does this mean? Like in the context of like decisions we have to make or like things we have to communicate to clients. Like, like how do you translate this like data thing to an actual like practical real world thing?
On teaching and AI's impact on learning
Yeah. I had a similar, similar experience. Like just teaching that kind of intro to programming and data science just felt so empowering to people. And that's, I kind of feel like that's, I do feel like, I don't know, wistful that it feels like the next generation is not going to have that experience just because it seems like so much of what they're going to do is, is mediated through AI and that kind of pure experience of like turning your ideas into code that the computer then does something with. Like there's just such a, like a fun, like, yeah, it's empowering. It's something that I love doing. And it just feels like that's like that relationship between you and the computer has fundamentally changed it. Like who knows where we're going to end up, but it certainly seems like writing your ability to write code is like less important now than it was five years ago. And I, yeah, I do feel like wistful for those simpler times. I admire a lot of the educators right now who are trying to figure out how to adapt, you know, the way that you teach in the wake of like, we do things differently now, you know, not everyone does, but I do, I do things very differently now. And I think people learning are going to be doing things very differently. And like the way that you teach has to change as well. And that's not always something people are prepared for.
But I guess, I mean, people must have had some more like wistfulness about like, oh, I was so good on the slide roller. Like, and now like that skill is just, I felt so empowered when I learned how to use the slide roller. The punch cards, the punch cards. Yeah.
Yeah. It is an interesting trade off. Do you have, do you have a sense if you had to like teach someone how to like get started with like AI and data analysis today, like how, how would you approach it?
Man, that's, I think about that sometimes. Cause like I was kind of like, kind of when I was like wrapping up at the University of Utah, they were starting up like AI initiatives and like doing all of this kind of stuff. And I'm like, I don't a hundred percent know, like I have, I think I have things that I would try. Because like the way that I learn now is different. Like I've always taught people in the way that like, like I, I try to remember, like, how is it when I learned something and like use my own experiences from learning to like teach. Right. So I think like now the way I learned something, and I probably would try and teach in a similar way is like, I try and get the basics down. Like sometimes I use AI to do that, but I'm not trying to like use AI to do it for me until I feel like I at least understand the general syntax of the language or like whatever I'm trying to do. Like, like let's say Streamlit was the example. Cause I literally like had never heard of Streamlit and I had to use Streamlit, which is like essentially a dashboard, like a reactive dashboard. And that uses Python code. It's a Python library.
And like, you know, I just kind of went to AI and I like, I think I went to Gemini and I was like, I need to create a Streamlit app to do, to create this plot. And I just looked at the code that it gave me. And like, I at least have like, I know how to code. So like I can kind of like look at that and be like, understand at least somewhat what it's doing. And then I'll kind of like go through and I'll be like, what, what does this line do? And what does this line do? And like, why is it structured like this? And like, I'll try and like create something and copy it and run it. And then I'll be like, I'll see what errors I get. And like, maybe I'll do some like regular Googling or AI Googling, you know? And like, I'll, it is very much an interactive thing with AI. Like I'm not, I don't start out being like, like do it for me and not expecting, like I start out maybe being like, do it for me as a learning experience. Like do it for me. But like, I want to understand what you're doing, you know? So like I do use the AI kind of like as a tutor and like very quickly, I feel, I felt like I understood kind of how the syntax worked and like, I felt like I could probably like modify it. I couldn't necessarily write it from scratch myself, but I don't know that I need to be able to anymore now that I have AI, you know? So it's like, like I think learning for me, like I want to get to a basic level of understanding so that when I do encounter errors and want to do some more sophisticated things, I know how to guide the AI. But I do actually use AI these days to get me to that point. Like, but I don't necessarily vibe code as like, say, I don't care how this works, just do it. Like, I don't know. It's a trade-off. It's a balance. So I think like if I were to teach it, it would be like, how do we interact with the AI in a way that we can effectively understand what the code is doing and be able to at least conceptually create it ourselves, even if we can't necessarily write it and start to finish ourselves? Because I don't think we need to be able to anymore, necessarily.
when I do encounter errors and want to do some more sophisticated things, I know how to guide the AI. But I do actually use AI these days to get me to that point. Like, but I don't necessarily vibe code as like, say, I don't care how this works, just do it.
Yeah. It's really interesting to hear it's like generating a case study for you. Yeah. That you're able to kind of like take the example and have it break it down and then try to like transfer it to what you want to do is like a learning aid. So you like pick up a little bit, but you're also not needing to start from like zero.
The question I kind of wonder about is like, do you need to be able to do like maths in your head when calculators exist? Like how important is it that you still have some like programming abilities?
Yeah. I kind of think like at the moment it's still a little bit important, but down the line as these tools get better and better and better, like maybe less and less important. I think we do still need to be able to code, but like, I don't know how long that'll be true for.
Will code even exist?
There is still, there's always still like the, like the imprecision of English. There's always going to be cases where you could look at the code and make that, like, if you understand it, you can just be like, oh, I need to change this. And it's really easy. And if you've got to express that in English, like it's either like it's ambiguous and you're going to go through multiple loops or yeah, as you said, sometimes I just get stuck doing like weird things in an endless loop and it's just easier for you to do it yourself.
But yeah, although I do kind of wonder, like, I mean, right now it's like we use the AI to write code or the AI is writing code under the hood. And that's like, like at some point is the AI going to get so good that it doesn't really need like code. Like the code is like just a tool we use to get from like question to answer. Right. Like, like, is it going to develop something where it's like not really code as we know it, you know, like it can do the reasoning and figure out stuff, but it's not necessarily writing Python code or R code to like answer the question. I don't know. Like it's like this kind of formless thing in my head. Sometimes I'm like, like the code is the tool that we created for us, but like, maybe at some point, like for certain types of tasks, like the code itself is kind of going to become obsolete. I don't know. Yeah. It's a tricky one. I feel like with arithmetic, it's funny because like arithmetic, like the operation of adding numbers is one a calculator can do, but I almost feel like the code is the like, if you were to break apart that sequence of numbers you're adding, often it's still useful to have that. That's like more the representation of the problem. Like if you're accounting and you're adding up numbers, I feel like having those numbers there is to me the code where the adding is kind of like the execution, like still having that representation seems useful, but I guess it's also fair, like at a point where you could just, you're confident enough, you could just use English and not even have to think about it. And the point is like the, but AI is never going to be as reliable as like a calculator in the sense that we can prove that a calculator can do addition 100% correctly. Like AI can never get to that point.
It still isn't very good at adding. I think I just wrote a blog post about LLM struggling to add numbers. They've gotten better, but you know, it's not 100% of the time.
Oh yeah. Like I still, I feel like when I Google something and it comes up with that AI thing at the top, I just usually scroll straight past it. Cause I'm like, it's just wrong. Like it's 70% of the time it's just wrong. Or it'll be like, according to one Reddit user, it's like, is that your source? Yeah. It's pretty freaky. I think we're getting, we're getting pretty long in the tooth kind of running up at time, but I, yeah, I think I really appreciate just all of this like thought on AI and teaching. And I'll say, I am impressed that you have like held onto your accent for pronouncing data. And my accent, I think is at this point, it's increasingly American though. Like there's like, I think you still sound like a solid Kiwi. But I don't say data. I say, I don't say data anymore. I say data. Yeah. But it's data. I mean, I, when, when I was teaching R a lot and maybe you get the same thing where I'll just be like, ah, and people are like, yeah, that's why I say R now. That's because Australians and New Zealanders have a non rhotic R, which is set like the vowel sound. It's very confusing. I like to think that you're, are you saying whenever you chat with like an Australian or New Zealand, you're like clocking their use of data. You're like. Hadley clearly is. Yeah. Just when I'm in America and someone says data, I'm like, oh, that's not an American. I think there's some words, like I do put R's in places now where I don't think they belong. The letter R and the sound R, but I like, there's certain words that I hold onto with dear life. So for example, like I'm not wearing a sweater right now. I'm wearing a jumper and I will hold onto that for the rest of my life. Even though every time someone's like a jumper, isn't that like a jumpsuit? And I'm like, no. I love that. I think it's good to keep people on their toes. You know, like you got to have like a couple of things. Yeah. Rebecca, thanks so much for coming on. I really appreciate all your thoughts on AI and also your role building out tooling. I feel like it's so such an interesting role for data scientists to take in orgs. So thank you so much for coming on. Thank you so much for having me. It was good to chat with you all. Yeah. Thanks Rebecca. Thank you. Thanks for coming.
The Test Set is a production of PositPBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio, Agi. For more episodes, visit thetestset.co or find us on your favorite podcast platform.