Transcript#
This transcript was generated automatically and may contain errors.
Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
On this episode, we sit down with Nell Thomas, who has led data teams at Etsy, Facebook, and the Democratic National Committee, or DNC. And now she's VP of Data at Shopify, which has mandated that AI usage is a baseline expectation for every employee. She leads a team of over 400, and she'll be the first to tell you, nobody knows the right way to do any of this yet with AI. We talk about the modern data stack, building trust in data organizations, and why the smartest move right now might be resisting the urge to lock anything in with AI, but embracing exploration and discovery.
All right, Nell, welcome on to The Test Set. So I should say, you're Nell Thomas, and Vice President of Data at Shopify. But I also have to admit, I'm totally enchanted by your career, I don't know the word for it, record choices. So you were Vice President of Data, or you are at Shopify. It sounds so official when you say Vice President.
I'm so sorry. I didn't know. It's like when you meet a dignitary. Yeah. So I was, I just get on one knee. And before that, you were CTO of the DNC, Democratic National Convention. Committee. Committee. I'm so sorry.
No worries. We got this. We're good. We're good.
And before that, you were at Facebook. Yep. Yeah. Before it was Meta. Yeah. Nice. Before it was cool. Or when it was cool. I don't know. Was it ever cool? One of those. We'll figure it out. And then Hillary for America. And Etsy. So I know, I know it's a long. I know. And there's like even, you know, time before that. It's a nice way of saying I've had a long career. Yeah. Lots of interesting choices. So I'm excited to talk about them. Excited to be here with you guys.
Yeah. Thanks for coming on. That's all to say I'm enchanted by your weaving between sort of like political impact and industry work. So maybe just by way of introduction, I'm Michael Chow and joined by my co-host, Wes McKinney, principal architect at Posit.
And we're in Times Square. So there's a very large glowing billboard just rocking behind us. Just to make it more complicated for you to get distracted. It means I have to up my game so that you, I can compete with the Times competing with some kind of ad for something. But thanks for coming out. Yeah. I'm so excited to talk a little bit about your career and this really interesting path that you've woven.
And also about your work as a leader of a data team and a data group. And I think in 2026, that often involves the emergence of AI and what that looks like for folks. But maybe to start, maybe you could just tell us a little bit about yourself. Yeah. And yeah, I'm very excited to be chatting with you both. So thanks for having me on. I've been working in data now for two decades, which is a crazy amount of time. And I'm probably could start the conversation at like any point of that journey. But let's say like from, to start at the most important thing, like I love working with data. It's always been something that's motivated me. I love finding interesting, hard problems. And I think I've been very lucky to have this opportunity to sort of haphazardly, like make my way through a series of different choices.
But before we get to the career stuff, like I'm also human. I live in New York. I live in Brooklyn. I am a mom of two little girls, which is an important part of my story right now because they are relatively young and take up a huge amount of energy and time, which is exciting and awesome, but also real. Yeah. And at Shopify, I run a team right now. It's a little bit over 400. It's data infrastructure, data engineering, data science, some analytics in there. And it's a really fascinating place to be working. I joined three years ago and really been following the arc of the advent of LLMs and AI and how that is rapidly evolving and thinking about how I run and manage a large team. So happy to touch. I can go back on the career in more detail.
Building data organizations across eras
Yeah. Well, I think what's interesting is that you've been working as a data leader over essentially many eras of big companies figuring out how to do analytics and data science in a way that's sustainable and scalable with good best practices, like engineering mindset. And so I remember I first got involved in the data ecosystem in the early 2010s. And so that's when people were starting to talk about big data. And we heard about large data orgs and data problems at companies like Google and Yahoo and Facebook and so forth. But then all the rest of the companies in the world were like, OK, we need data teams, too. How does that work? What software should we be using? Who should we be hiring? What is a data scientist? What does it mean to do analytics and to build analytics teams?
And so I think you've had a front row seat to doing that inside both really interesting e-commerce businesses and as well as tech and tech and politics. And so I think you have a pretty interesting perspective on how that understanding of how to build an effective data organization has evolved. And I'm sure with each passing year, especially now, it's changing rapidly.
It's funny because it's a weird combination of things changing rapidly and still the same problems at the heart of things, right? The quality of your data is always the number one problem, right? And now maybe more so than ever. But yeah, it is this interesting, both two different lanes of things. Some things are just consistent backbone of core issues while we're seeing the tooling and the skills kind of rapidly move.
So I graduated college in 2005. Before the word data science was even officially coined as a thing, and I started work. I'll just make a very brief note here. So I graduated with a degree in cognitive neuroscience and psychology, and I worked in psychology labs running experiments on humans. And that involves obviously learning a lot of stats and doing a lot of work with and analyzing the data that comes from those experiments. When I graduated from college, all I wanted to do was move to New York City, obviously close to Times Square. And so I took basically the first job that hired me, and the thing I had that was marketable was my stat skills. And so I took a job in equity research, which is a variant of finance, where I was doing large field data analysis to predict the movement of stocks, basically.
And what was the job called? What were they calling? Yeah, it was called a research analyst. Nice. Okay. Yeah, yeah. But as an undergrad, I used SPSS and Stata and basic familiarity. When I started at this job, they basically handed me a book on, this is how old it was, the book, SQL for the workplace. And that was my first job. It was learning SQL. Python came much later in my life, but it was very much on-the-job learning. And this is what it means to extract value from large data sets and make it meaningful for some outcome. And that was way... In essence, it's a lot of still what happens and what the goal is, but way before we had some of the terminology we have for now. And right before big data became this craze, and you have this emergence of the crazy data industry that cropped up, I would say, within four or five years. But it set the ball rolling in this sort of unexpected way where I actually found something that I loved to do unexpectedly. And then it took me a few curves to get to the rest of the journey, but I never would have anticipated it when I was an undergrad that that's where it ended up.
Yeah, not to try to go too deep into it, but I'm really curious about when you... So you mentioned you did social psychology and neuroscience, and I'm really curious if when you started that, if you were planning on becoming a research analyst. Absolutely not. I mean, I studied those topics undergrad because they just were really interesting to me. I think in my head, I always assumed I would go back to grad school. And so I was like, oh, out of college, I just want to get a job and be out in the real world and live in New York. And I wasn't too fixated on the topic as much as I was getting out and experiencing working. And so it was pretty... Random is probably too strong a word, but it wasn't premeditated.
But again, what's funny is the through line now, in retrospect, is I can really talk about, I've always been really interested in human behavior, understanding it, using data to understand it, which is honestly at the core of a lot of psychological experimentation work. And so, yeah, there is more of a through line than I probably realized at the time.
Yeah. I would imagine, I'm curious to hear some of this journey, but yeah, running a 400 person team is maybe very psychological in a lot of ways. Oh, 100%. I mean, management and leadership is obviously a lot of dealing with humans, at least right now. There's a lot of understanding how to help people do their best work. Now, I think a lot about my job right now is creating scale and leverage. So how do I create systems that enable people to do phenomenal work? And that means making sure that no one is limited in what they can get done, but that we're also holding a really strong quality bar for everyone at the same time.
And so it's kind of, you know, it's actually, it's a little bit when you think about it from a data perspective, it is a little bit like law of large numbers. Like how do you make sure that you keep raising the standards while not putting caps on that would reduce quality for anyone? And it was a lot where I spend my time is also just making sure that, you know, people feel like they are energized, they're motivated, they have the tools they need, they have the context they need, and that there is good scrutiny on the work.
Because actually, I think that's like the number one thing I've noticed is in organizations where people aren't actually paying attention to the data work. It's very hard for it to be amazing.
Because actually, I think that's like the number one thing I've noticed is in organizations where people aren't actually paying attention to the data work. It's very hard for it to be amazing.
Oh, interesting. Because you need like a great audience to appreciate it. Yeah. And ask questions and to push it and deepen it and challenge you. And so that's like another thing that I always want to make sure is happening.
Yeah, that's I feel like the idea of scrutiny is such an interesting one. Like, I don't know, in an org, what does scrutiny look like? What is that?
You know, it's also it's kind of a bad word, right? Like, I think you hear the word scrutiny, you're like, yeah, yeah, it feels like it feels like, So I check my work, but don't look too closely. Exactly. And like, no one, I mean, no one wants to feel like their work is constantly being questioned, or like, that's not a good feeling either, because it suggests a lack of trust. Right?
Yeah. And like, people who are have amazing skills and are curious and are great problem solvers, you know, they don't want to feel like someone's like coming in being like, but did you like those things together correctly? And, you know, so it's not that type of scrutiny. It's more like, it's more like people who can appreciate it, like maybe like thinking about like having a really great audience for, for like a play. We're doing a play with no one there watching it. Like, are you going to do your best work? Right? And but it's not just people clapping. It's also people like engaging with the material and being like receptive to it and like contributing back to it in a way that has a really has the ability to like, to make someone feel like they want to do better and better work, right? And they want to like, keep staying in that conversation. So I think it may be scrutiny. I mean, I should think about a better word than scrutiny, but it I think it's about attention. Yeah. Anyway, you asked me what it looks like, though, I would say it's making sure that that there are the right forums or rituals could be meetings could be like async things, but where people's work can be like reviewed and discussed and debated and and is part of the conversation.
The essential role of data in modern business
I mean, I feel like what's what's part, you know, partly so interesting about modern modern data teams is how how essential data has become to every every aspect of the business and how businesses function. And so I remember 15 years ago at conferences, there was this discussion about like, you know, every company needs to become a data company. And I think today, you know, the companies that are left are all effectively data companies. And so using data and making data, but both collecting the right data and then processing and organizing it and making it accessible in ways where it can be available to all of the people need it and get the results, you know, the analytics and the the, you know, the insights from that data in a way that is timely and actionable so that it can influence decision making can be a part of product development and like all the decisions that get made in the business.
And so there's probably never been a time, you know, greater than now that businesses have been more more dependent on their their data teams to be able to function. But that also like, creates this tension where like the entire organization is like, like our data team, our data team, like, you know, they don't have something they need, like, they're not able to get the answers they need fast enough, something doesn't work quite right, like the tools that they're using to analyze the data, you know, the dashboards or the query interfaces or whatever tools they're using, if they're not, you know, working in the way that they need to, that does make the business run worse.
Yeah. And so that that does put a lot of, like, you know, pressure on the data organization to deliver and to continue to innovate and, and do better and better to make the ultimately the better the data team does, the better the organization can operate, and that that yields more success in the future. And so I'm curious, like, how you see that playing out? And yeah, how you manage kind of those, like, the expectations that the rest of the, the organization has on what your your team is building?
I mean, that was incredibly well said. And I 1000% agree that fundamentally, like, most of the great companies right now are data companies, and how well they leverage that data and the systems they build around it are one of the things that differentiates.
completely agree. Yeah, I mean, some, you know, I like to think about the kind of the the value chain of work of data from from raw data creation, all the way through to, like, how that data is being used. And one of the nice things about my current role is that, especially leading the data infrastructure team, I get to be pretty far left of on that sort of my spectrum, this is left of from raw data creation, like how that data is ingested, how it's processed, how we make sure that it is kind of prepared for downstream users, both for analytical purposes, or like kind of data science purposes, and for production purposes. So for building ML models, or, you know, any LLM use cases. And that so kind of being really far left on that spectrum, is a place of immense leverage, because the better we can create really well, a make sure we're collecting the right data, which is always a fun problem. Again, one of those age old problems that's been true for my entire career. And making sure that's captured in a clear way and making sure that we're actually being cost efficient. That's increasingly an issue now that we see like how quickly costs can increase around LLM usage.
All of that creates this ability for the whole company to operate better, like you said, whether that's the ML models being built, or the analytics are being pulled out of it.
I think in terms of managing the appetite for that, which I think is the heart of your question, I see that as an incredibly exciting challenge. I'd always rather be working in that environment than trying to convince people why data matters, right? Which at various points in my career, that has also been part of the job. I think that right now we're in a moment where I don't need to convince anyone that they need to care about that spectrum of data. I think it's more managing the reality of what is possible. Managing like the expectations around how quickly we can get what we need, and making sure that we have the strength in our foundations on that kind of left side of the spectrum so that we can trust the outputs. Because increasingly, as you see, it's very easy to work on the right side of the spectrum with vibe coding and lots of ways that you can very quickly do an analysis. Underlying data quality, underlying foundations make everything, it's only positive benefits downstream from that.
And so I'm really orienting myself and my team around making sure that is where we have the trustworthy work so that we can make it self-service, we can make it accessible. We have an MCP for our data warehouse, or anyone can like, you know, very easily access that data. Making sure the right privacy and security controls are placed to make sure that's all guarded in the right ways. Like those are where I'm like really passionate right now because I think it creates, it does manage a lot of those things naturally when you focus on trust and quality and systems.
And are you saying, it sounds like your side of things is very like data engineer heavy, is that right? And analytics engineer heavy?
Yeah, yeah, yeah. All of the above, like yes. Yeah, we have a relatively sizable analytics engineering team that has traditionally focused on the pipelines that we use for all of our data science work and kind of business analytics reporting work, dashboarding, etc. We also have data engineering problems that are much more about using that data for some of our production use cases. And then again, that's the fun thing about being a company that's at the scale of Shopify is that you have like great use cases on all sides of this, whether it's like how we're, you know, using data that we're getting to understand, to like predict likelihood that a visit is a bot and whether we're using that either for routing that traffic on the edge or whether we're using that on how we are counting visits, right? Those are like a very wide spectrum of use cases. One is for infrastructure purposes and one is for, you know, business reporting purposes, but it's actually the same data. It's the same model. You don't want to duplicate all that, right?
And so that's where having, you know, having that holistic view of the sort of data journey and the types of data work happening creates a lot of value. And I've always felt like data is a very ambiguous term. So you'll say like, I'm a data scientist, like what does that mean? Like what part of the data stack do you work on? And so I'm more and more seeing like, there is a lot of differentiation in that spectrum, but I think that people who are really good have a, can kind of run that spectrum in their head at least. And they can see the connection between, hey, this is the raw signal that we have over here and here are the myriad use cases. And they can create kind of unified solutions that help enable all of them. And so it's one of the things I definitely look for when I'm like looking for new additions to the team, is like someone who can, they don't need to be a specialist in all of them, but they can, can they see that.
The data value chain and AI disruption
Yeah. Yeah. I mean, I think like one of the, one of the problems that I, not to steer the topic and still want to kind of stay on this, this kind of platform kind of architecture problem a little bit, but problem I've been thinking a lot about lately is like, you know, we think data teams do all this work to, to build kind of engineer this whole pipeline of like data collection, curation, ETL, data, data cleaning, data quality, and then engineering the data warehouse, like designing the storage, the databases, the data access layers, and then kind of the metrics engineering, like semantic modeling, you know, so you get further and further up the value chain. And eventually, eventually you start reaching the users and you can start thinking about like, how are they querying the data? Like what their dashboards look like? Are they using Tableau? Are they using some other business intelligence tool? Are they using Python? Are they using R? Like what's their interface?
And so I feel like right now, what's really interesting is that we're seeing AI pop up and people using LLMs, like in all of those different parts of the value chain. And it also, it's, it creates a lot of opportunity, but also like there's, you know, in every place there's an opportunity for like, you know, agents gone wild a little bit. And so, you know, I can imagine, you know, there are going to be instances where the end user who, one thing that people are talking a lot about these days is this concept of, you know, you're probably well familiar with it of like headless BI. So like, like who says I have to use, you know, the Tableau's UI, like just give me the endpoints and I'll use cloud code or I'll use, I'll use chat GPT to vibe code my own, you know, custom dashboard and not knowing that what's inside might be some very large SQL queries that the person building the, building their, their personalized dashboard, like, you know, they can't read the SQL or they can't judge whether this pile of hairball of vibe coded SQL has errors. And so they get a dashboard, they're like, Oh, it looks right. It actually contains errors. And so it's kind of a little bit of a disruption of like, you know, the more heavily curated, like actually having, you know, an analytics engineer or somebody with data science experience, like building the dashboard and writing the SQL by hand and making sure that it, it, you know, what, what you're seeing is like what the reality of like counting things correctly.
Yes, I agree. But I'm also going to challenge that a little bit because I think that, I mean, I've observed at a lot of companies, a lot of like overly complicated verbose SQL with business logic and business logic embedded at that like last mile delivery of a data dashboard as well. Like, you know, it's very easy for someone to, I don't want to throw any particular BI tools under the bus. So I'm like, I'm not like saying I'm not naming names, but it's very easy for a lot of these tools for you to like actually push your, instead of doing your semantic modeling in the right place and doing it in a way that's code controlled and doing it in a place that you can really audit, they're, they're putting it like in basically for the presentation. And that becomes very inscrutable. It also creates like common problem at a company is we have 1000 definitions for this core metric. And you have to, you actually have to ask the person who knows the right SQL query to write, to get the like correct value. And again, that can all, you know, best practices, you're handling all that. But you know, a lot of companies like that, there's like slow creep. And all of a sudden you have like, which is the right dashboard to use for X and like this person gave me this number and that person gave me that problem number.
So I don't think the problem that you described of like, can you trust the dashboard is new with the emergence of AI. That's totally fair. BI tools can get very creative, I guess, with the SQL they generate.
So I don't think the problem that you described of like, can you trust the dashboard is new with the emergence of AI.
Yeah. And I think, I mean, it's, I think it's a not uncommon, but suboptimal value prop that sometimes the data scientists have, which is like, I know this table the best to write the right, it's not even like knowing that it's not knowing the code. It's like, I know the nuances of this particular table in a way that's like arcane. And I, I'm the only one who can like properly write the where clause that suppresses the types of things that shouldn't be counted in a way that's like, you know, and I think that that's not ideal because then you're encoding a lot of really important information in like one person, as opposed to making it something that a whole team can benefit from, right? Like, how do you get the system, how do you get the scale? So I think everything you described about like how to make sure that you have, you know, really great, you know, ETL, really great data, you know, pipeline orchestration, really, really strong modeling, have semantic layers, have you have, you know, the right tests to ensure that all those things are running well, like so, so, so, so important. You know, I think that humans and LLMs might be like almost just as likely though to like kind of have some challenges and get and making sure that you have that like consistently trustworthy presentation. Not that I trust LLMs more than humans, but it's more like it's a common trap that everyone else.
Mapping the modern data stack
Right. I almost wonder if at this point it'd be useful because we've talked a bit about like data engineers, analytics engineers. Yeah. A little taste of BI and this kind of like big stack. Yeah. Would you be up for almost trying to lay out who's involved in this? Yeah. Stack as you see it? Yeah, yeah, yeah.
Sure. So, I mean, I think, you know, all the way on the left, it's usually you have your, like in my mind, this is just my experience, you have your production engineers who are like writing the code that hopefully generates the instrumentation that gathers the data you need, right? Whether that's writing to a database or firing an event that, you know, is sent off to some sort of log. You know, then you have your infrastructure, your data infrastructure engineers that are ensuring you're ingesting that data correctly, right? So whether that's how the events are being fired into whatever Kafka topics or whether it's your replicating data from a database and then doing the work and building the data platform of processing that through a whole bunch of storage and compute layers that are doing some sort of modeling of that data, right? And like usually, you know, at some point you're handing it over from like what is more of a data platform engineer of like creating the house to like the data, the analytics engineer who is then ensuring that the raw ingested data becomes model data that is hopefully some sort of canonical data asset that then the data scientist can query.
Yeah. And just to check, is it at that point, like once it hits the analytics engineer, you're in something like BigQuery or...
You're writing a sequel, you're writing a DBT, maybe you're using Airflow, you know, I mean, that's like the stack we currently use, but yes, that's very common. Maybe you're using Snowflake instead of BigQuery, but obviously flavors vary. But I think that sort of those big buckets are pretty universal. You have the team that's responsible for like maintaining the data house, and then those people are like working inside the house, as I like to say.
And then obviously at the very end, you have your presentation layers, whether that's Looker Studio and Google or Looker or, you know, there's some fun new ones, but, you know, whatever flavor of how that looks, they all have their limitations, in my opinion.
The other use case here that I kind of mentioned in the beginning is also like the diversion of some of that data out of the data science use case into sort of more of the production modeling use case. So like, hey, we're going to model this data to use it for recommendations, for search, for some sort of prediction, right? And those cases, usually the handoff isn't to an analytics engineer, it might be to a data engineer, to a machine learning engineer, kind of the path of coding for the machines as opposed to like coding for human input, which is where I think usually most data science and analytics work ends up, you know, it's something that a human looks at to review and make a decision about as opposed to like a machine is taking it in to decide about how to rank something or how to display it.
Are you saying like the analytics engineer is modeling the data for like an analyst or people downstream to kind of like pull it, but once you get into like special cases of like modeling where it kind of requires like domain expertise or like modeling skills?
Yeah, I mean, I think the two, to me the differentiation there is a little bit, like one is latency. So like analytics engineers often working in batch, you know, might have, hey, it's totally acceptable to have a 8 hour, 12 hour, 24 hour delay and your data being fresh because you're counting things to, you know, do an experiment analysis or do a user behavior analysis of usage of a product or a feature changes over time, but usually have a longer scale.
Some of the production or operational use cases I'm talking about, you might, you have very low latency requirements, right? Like I want to be able to process data very quickly because I want to see what a buyer is doing on the site to be able to change what I'm showing them on the site, right? And so that does, has different requirements for the data infrastructure than you would for the, you know, analytics use case. You know, I think what's interesting, if I'm like driving this fork in my head, I guess one of my points here is that like the further left you go on that, the more unified it should be. And then it kind of starts to fork over time, but you don't want those forked the whole time because you end up with a bunch of data replicated, a lot of costs involved, and probably a lot of like lost knowledge and context and value because you're not getting like great reusable learnings between those two streams.
It's so helpful, I think, to lay out the people. Like I know it's a lot of work, but it's like so helpful to hear it kind of laid out. I feel like oftentimes there's a lot of data talk in it. It's hard to like keep track of the roles. Yeah. I feel like it's really helpful to hear. But the other thing I'm really curious about is we just talked with Tristan Handy from dbt labs recently who just tried to break down like.
Very much so, yeah. Like, what was that world like versus the world today?
It's a really good question, and now you're going to test my memory, too. No, no, I mean, Etsy's an important part of my, like, Oh my god. World today. So, it's a really good question, and now you're going to test my memory, too, because —
Yeah, it's hard to send you back. No, no, I mean, Etsy's an important part of my, like, I think my evolution as a human, but also specifically as a data person, because it's where I learned to be a manager and grow a team. So it holds a very special place in my heart.
Also very quickly on the, like, laying people out, I think my number one recommendation to any person who's, like, new to a company is to get your hands on an end-to-end data flow diagram. And like, what are the technologies being used in every stack? And who owns them? And it really, I don't know, I think it's, to my point earlier about just, like, having that holistic view, I think it really helps connect dots about, like, where you're, like, it's kind of like a food chain. Like, what are you, what are you eating?
And are you, are you telling people to Google the term modern data stack or something? No, I don't want to divert you from Etsy. I feel like you brought us there. But I mean, it's, you know, because every company has their own flavor of it, you know? And it's like, you're going to, you're going to end, the teams that own it are going to be a little bit different. The terms they use, like, all that. But if someone's bought, they're like, now I'm in, I'm ready to look at this diagram end-to-end. What are they typing in? To that, well, I mean, you, I think this is, you should ask someone internally. Like, at a new company, be like, hey, like, do you have this? I bet half the time people don't have it. But if they do, like, hey, we're, like, or they only have part of it. They're like, oh, yeah, well, the data's in BigQuery. And then we use this. And you're like, well, how did the data get in BigQuery? Like, you know? And then it's kind of like asking that infinite set of questions. Like, well, where did that come from? And where did that come from? Where did that come from?
Just tracing every data point from, like, you know, somebody clicking on a website to like, how does it, how does it end up in this dashboard? How does this event get fired? Like, I literally, like, you can see events get fired, right? Like, oh, this event got fired here. Like, what, how long does it take to go through the different hops? Where does it go through? It's a way to, like, make tangible, also, some of these things we talk about that are super abstract. Like, oh, the data was, like, moved from here to there. And then this team does this. And you probably find, like, I don't know, you learn something about the data you're using along the way.
Navigating technology choice
Well, I guess, like, a closely related question to that is, like, how you've seen, like, you know, not only Shopify, but also, you know, kind of other teams that you've led just, like, navigating technology choice. Because I feel like as time goes by, as time goes by, it gets harder and harder. And, like, the landscape, I feel like each year, like, you know, it's like we got to zoom out a little bit. And so now we're at a place where, like, the modern data stack, you need a microscope to, like, look around at, like, all the different thousands of technologies that exist. And, you know, there's open source projects. And there's companies and products and things. And, you know, I remember there was, like, a famous blog post years ago, like, choose boring technology. And that's probably, like, the right answer most of the time.
And a lot of times I can imagine, like, engineers are enthusiastic about, like, some new open source project that they discovered on the internet and using that thing. But, you know, often, like, adopting new technology, any new piece of technology you adopt carries, like, risk and maintenance and, you know, institutional knowledge and all that. So it's a big topic for sure. But, yeah, curious. Oh, and it's a huge topic. I mean, the question of there's the classic, like, build, burst, buy part of this, too. And there's the deprecation of legacy tools, which is really important, often doesn't happen. And there's the sort of fragmentation of the technologies people are using internally. And, like, how much do you, like, encourage and permit versus how much do you clamp down on that?
I remember when I was at Meta, it was, like, people used all sorts of different, like, personal data stacks. And the data sources was the same. But then some people were using Tableau and some people were using, like, Modes. Like, it was kind of a wild west of, like, what, you know. So I always thought that was kind of fascinating. But I'm going to attempt to sort of answer your question by connecting it back to your question about Etsy.
Which is, when I was at Etsy, it was, Etsy was very big on, like, do-it-yourself. I mean, kind of per Etsy's vision as a company. Yeah, it's very DIY. It's a very DIY company. It was founded by a carpenter. It's very much, like, about, like, you know, people building craftsmanship and other artisanal work. I mean, like, our, like, our database were all, like, on-prem. Like, it was very, it was very, like, you know, build your own stack. So that was true of our data as well. And we were writing, I mean, I was not writing, but, like, our data folks were writing, like, Scalding. Like, it was, yeah. I haven't heard that name in a long time. Nope, you haven't.
It did have a moment. It did have a moment. We hired, and I'm blanking on his name, but we hired the guy, the guy who, like, was a big Scalding proponent at Etsy. Oscar? No. Anyway, there's different ones, yeah. Or Abby, maybe. Yeah. But yeah, we, we had a Vertica cluster. This is where we did our, yeah, which is also an old technology. Weirdly, I've done two Vertica migrations in my life. One at Etsy and one at DNC. Like, older data stores. So, it was a lot more homegrown, like, a lot of the tooling generally. It was a little bit deeper in the stack, which actually, again, I don't mind, because I think it does make people, like, generally, when you have less sanitized data, as a data scientist, you're going to be getting your hands a little dirtier. I think you're going to learn a lot.
Yeah, obviously, pre-dbt, you know, people were writing a lot of, and some people were writing R, some people were writing Python, you know, people were writing, obviously, SQL. But a lot of the challenges we had there were not that dissimilar to, like, what I see now in terms of, like, where is the data for X? And is this the, am I using the right data? And I think with LLMs, those are the same, they're still the same questions, right? Like, to your point earlier about it, like, can I trust what the LLM is sharing? Because I can interrogate what data source they're using, and are they holding it correctly? And there's so much, I mean, there's, unfortunately, there's a lot of art still in, like, are you kind of, like, using your data set correctly? Because you understand the, like, nuances and arcane details in a way that means you can leverage it correctly.
And I think that part is, you know, it's still going to be, it's going to be an interesting evolution of the challenge for us to figure out how to make sure we interrogate the way LLMs are using it and create kind of visibility on the choices and, you know, an agentic data scientist is making and how they use their data and make that more transparent and get more testable. And, you know, we can quantify some of their, you know, errors, etc.
Agents and data platform load
I mean, one of the things that I, you know, things I'm seeing discussed a lot lately is agents putting a lot more load and stress on data platforms, because essentially, like, you know, used to be like, you write a SQL query, run a SQL query, you look at the results in my list. So essentially, like, often, like the, you know, human in the loop, using the data platform or, like, using a dashboard, kind of limited by, like, you know, human in the loop doing stuff and thinking and using their mind to think about things. And that kind of creates like a natural, like, kind of rate limiter in terms of, like, the number of queries and so forth that are being run. But now you can have, you know, a bunch of agents, like, interrogating a data platform. And so, you know, in a way, it's almost like not differentiable from a DDoS attack in some cases, where it's just like, I run the SQL query over and over until I understand the problem.
And so probably that, I imagine that's going to change, like, the way that the data platforms are designed and with guardrails to help the agents, like, not DDoS the data platform, but also, like, still to be able to, like, be compatible with their needs in terms of, like, an efficient agentic loop, are they able to, like, efficiently get access to, you know, also, LLMs are not good at looking at large amounts of data. So you also, like, can't give them very much data to look at at any given time.
Well, again, I think this is where some of the best practices that the industry has been talking about for decades are still the most important best practices, which is, like, you know, work out in the open. Like, when you're a data scientist and you're writing code, first of all, write code. Don't do your analysis in, like, Excel. So, like, write code. Like, check it into GitHub. Make it, like, observable by others. You know, have it code reviewed. Store the output of your analysis in a notebook. Like, have it all be visible and accessible so someone can walk through your chain of logic. Like, what does the data use? What were the assumptions made? How has the data changed or manipulated to reach some conclusion? I think that, you know, we need LLMs to be able to follow that same path so that we can follow that, you know, we can follow their reasoning just as much.
Like, I will, I do get, I get shared on screenshots of cursor outputs all the time. Mostly by non-data scientists, right? You know, it's like, you know, I have a, I love one of my fellow VPs of the company, but he loves to do a rogue analysis and send me a screenshot and be like, I found the number. And, you know, my first question is always like, what data source is it using? Like, can you know? And I think this is where the more we can get, you know, this is all emerging so fast, but the more we can get people on workflows and have their kind of agentic toolkit include some of those best practices that we all know and love, the more we can trust the output.
he loves to do a rogue analysis and send me a screenshot and be like, I found the number. And, you know, my first question is always like, what data source is it using? Like, can you know?
It's interesting you made a point earlier about like, yeah, the value, the value of an audience. And also here, it's like, it sounds like you're talking about also the value of like an internal like audience or like putting your work out so people see it. Yeah. I mean, it really builds a lot of trust, right? Like when you can show, I mean, because I mean, and to kind of the point you made earlier, Weston, I kind of like a little bit challenge you on where I feel like there are a lot of people, you know, a lot of people will just like produce an answer. Like, oh, we ran the experiment and here is the result. And you get like a slide and a deck and it's like you then you're forced to just rely that you're okay. Trust this human is like doing everything. But, hey, actually, if we can build a system and a set of tools where it makes all of that experiment work, like really easily auditable and like you can see it and interrogate it and you can like that, you know, inherently builds trust in the platform.
And it also allows like the data person to do, I think, like higher order thinking a little bit. So I feel like that sort of when transparency in your work is the default, it just automatically breeds trust in the results of that work. And it kind of frees people up from that loop of like questioning, like what assumptions did you make when you do this analysis? Or like what was the time period again? Or like, you know, what ways did you filter or not filter this audience? It bypasses all that so you can just actually have a conversation about what to do with it, right? Like, okay, now what do we do now that we have this answer? And so I find that part of best practices like really compelling because it's not just like governance for governance sake or like, you know, dogmatic best practices for best practices sake, but it actually like speeds up the rate of kind of idea to answer and getting us to like insight faster.
it just automatically breeds trust in the results of that work. And it kind of frees people up from that loop of like questioning, like what assumptions did you make when you do this analysis? It bypasses all that so you can just actually have a conversation about what to do with it, right?
Culture of transparency at scale
Yeah, I'm so curious. Do you find that people, so you have a group of 400, which is a huge amount of people to track, but do you find people are pretty quick and willing to like be transparent or do these things? Or is there some resistance? Like, what do you see in people?
I mean, first thing I'll say is that I have a huge amount of selection bias in what I know about. Like, this is one of my, it's one of the hardest things about being, I mean, overall, it's a joy and a delight to do my job and I feel like it's an honor. So it's a hard problem is like a wrong thing to say. But it's a challenge of being in a like more senior role is that people don't always want to tell you the truth because they feel like they should be managing up in some way, which is a reality. And I also, I talk to people who, you know, work for me directly the most. And I talk to like your kind of like average entry level data scientists the least. And so I'm always careful to how I answer a question like that because I'm like, I would much rather you had like, you know, someone from my team here to like talk about like how they think about it as opposed to like what I think they think about it. That's a very transparent answer. I'm like, I don't know. I'm like, I don't know if I should trust the data I have.
Yeah, I mean, I think that, I mean, I think one like just in terms of how to encourage a culture where people feel comfortable sharing and working transparently. I mean, to me, the number one rule is making sure that when people share their work, they aren't, the dialogue is like healthy and productive. So even if there's like an element of like critical review, it's from a place of like good human interaction. And like people don't want to work with people who are jerks. They don't want to work with people who are like, you know, tough or demeaning or like being like critical of work just to be critical of work. And they want to do it if it's like, if it's making their work better and it's craft. So I think there's like actually human interaction is really important. And the culture of making it feel like people get excited and have exciting and positive interactions when their work is up for like debate, as opposed to feeling like it's only an opportunity for them to get criticized. So like an obvious thing, but like I think it's probably the most important thing is that you have to like to be kind of vulnerable and sharing your work openly. You have to trust that it's going to be received with good intent.
Yeah, I could see that being a huge factor. Like how how it rolls out and like is it an environment where people can be transparent and like put their work out? Yeah. Yeah. Yeah. I mean, and it's easy for me to say that. It's hard for that to happen at scale, too. But I mean, that is definitely one of my corporate beliefs is that like, you know, you want that sort of like radical candor where people can be honest and say like, you know, say the hard thing, but also do it from a place of like good human trust.
Yeah. Yeah. I can't even honestly I can't even imagine at the scale of 400 data workers like it kind of boggles my mind. Like I'm not sure my mind's ready to wrap around an org of 400 people cobbling. I mean, I think to me it seems like, you know, there's the challenges is, you know, for leadership is creating like psychological safety and where people feel like, you know, don't don't punish the messenger. Like people feel people feel comfortable, like if they if they see a problem, they see something's not working well, like that. They you know, they feel like they can share critical feedback about, you know, something they observe or something that's not something that's not working without it being like it's just ultimately the goal is how do we have a successful organization? Like, how do we do how do we do better? It's not about like trying to point fingers or blame people or say like, oh, this solution that we built six months ago, it worked then, but now it's awful. And like, we need to fix it. And sometimes like, you know, there's this retrospective looking at at past work and saying, well, it's not working. It worked then it's not working today. And somehow sometimes people feel like that is a like a judgment on the work that was done before to say like that was bad work because because it didn't scale or because like it didn't sustain us to the present.
And now it's creating a problem that like, you know, that's a negative judgment on the work that was done before. And sometimes like, you know, if you know, you allow kind of, you know, anyone who makes anyone feel that way in an organization, I feel like that's that's bad.
You know, because again, like, teams have to continuously improve and like whenever something, you know, was good, was good, then not good. Now, like, let's figure out something better. Let's make it better and not be too attached to, you know, like, you know, sweeping things, you know, I think some I've seen companies and teams like allow things to like, like rod and fester and like, you know, code that bit rods or like systems that outgrew their usability or that start to become liabilities.
And don't get torn out and replaced because it's like, oh, well, you know, we can't do that because it's going to hurt. It's going to hurt somebody's feelings.
Yeah, these are a couple things that really resonated and one psychological safety. I mean, it's a little bit of a well-worn term, but I think it is the core of your question earlier, too, and how you kind of make that as possible for teams.
I think also sometimes people do get really like, like things become precious. You kind of start having these like tools that no one can touch, right, or or decisions. And I think one thing I actually really appreciate about Shopify, because it has a real focus on like our CEO founder, Toby, is very focused on like first principles and like that's what comes first.
And that it means that kind of every tool and technology and choice is kind of always up for discussion. And there's no kind of sunk cost fallacy of like, well, we can't talk about that because, you know, we've already invested X amount of time. So it's like the orientation around not like specific emotional choices around tools, but around just like, hey, these are the guardrails of what we want and how we want to get it really I think opens up more room for that freedom for people to not feel encumbered.
You know, so many, so much of organizations is, is like you're always dealing with the years that came before you. Organizational design is always a product of like past choices that often are random, not random, but they're often like unintentional. Like you kind of get into the weird patterns of, why does this team report over here? It's like, well, you know, four years ago, someone was reporting to someone and they got into a fight and then moved it or, you know, whatever.
Like there's always this kind of weird historical accidents in organizations and in systems. And, you know, so the more you can de-risk, like re-interrogating those, I think the better. And I think it creates like a much healthier conversation internally.
Have you seen big differences since I know you've worked at like really different organizations? Yeah. Like what, what's that willingness to like interrogate and kind of like switch up been like some of the different kind of like in the different contexts you've been in?
Yeah. I mean, yeah, it's a great question. I think that it is, I think that it is a culture that is kind of, it is a cultural like artifact. So I'm kind of meandering around. So a lot of, as a technology leader, you spend a lot of your time thinking about tools you choose to use.
But that cultural vibe is actually, I think, almost as important because you're going to make a lot of wrong tooling choices. But if you can get the right culture and you can get the right people, you can make a lot of mistakes and it's going to be okay because you can switch them out, you know? And so I think, you know, I think I've worked a lot of interesting cultures. I don't want to like call anyone out more than others.
But I think that there have been places where there's more freedom to like interrogate and there's less. I think one thing Etsy did very well in terms of its psychological safety was they had a very, very big on blameless postmortems. And that idea of like, hey, we're going to do a retro and we're going to like make a point of going overboard about never ever having this be about an individual failure. It's always been a system failure. And how did like a person get caught up in that system failure?
You know, that spread around the company as a way of just thinking about mistakes generally. Right. And so that was a really important cultural touchstone. You know, I think that right now at Shopify, like Toby's investment in just like hiring people who are super high agency, who have like a high curiosity, a willingness to learn and teach learning like a muscle, like not something you do once, but something you do like every single day. And that's actually the most important skill set.
Toby's investment in just like hiring people who are super high agency, who have like a high curiosity, a willingness to learn and teach learning like a muscle, like not something you do once, but something you do like every single day. And that's actually the most important skill set.
I think that creates also a culture where it's like, OK, we're not here because we're experts in Rust. We're not here because we're experts in Python. We're here because we're smart people who can like solve problems and like use all the tools are still supposed to do so.
Political campaigns and data scarcity
Yeah, I know. So I will say I like heard on a podcast one really interesting thing that you gave. I can't. I think it was about the DNC, but maybe you can correct me about this challenge. I kind of like almost like the opposite challenge of like political campaigns where they make a lot of creative use of technology and then they kind of like wind it all down overnight and like throw it away.
Which seems like almost the opposite side of this problem. Like innovation just gets like thrown in the garbage.
Oh, my goodness. I'm sure I said that when I was at the DNC. It's a really big issue in political technology. Yeah. So just like to pivot for that for a second. Campaigns are these like ephemeral moments, right? You have a campaign. It gets spun up. It lives as an organization and as a set of tooling choices for maybe at best if you're a presidential campaign and you make it through the primary, maybe it's 18 months.
But like more often it might be three months, maybe nine months. And so, yeah, it's a lot of it can it can be a lot of quote unquote throwaway work and throw away org structures and throw away like and that is really hard on the overall it's hard on the outcome of wanting to be able to learn continuously and like leverage those learnings like cycle after cycle after cycle.
So the next campaign can be better at talking to voters than the prior campaign. And that's actually so it's one of the things that there are a lot of great organizations in the political space that try to create that like consistency across those cycles. Interesting. DNC being one of them where it's like, hey, this is like a we can make an investment in our technology and our infrastructure that creates like a really strong foundation that these campaigns can build up on.
And instead of the foundation being like this, the foundation is actually like this, where every year the foundation is getting better. And so the starting point is getting higher for every campaign. And that ideally we take the best of the campaigns and build them into the foundation and create this like really positive feedback loop. Yeah, that's the dream, you know.
Exactly. That's the problem. You're like so underwater. And the problems in political data, I think, are much harder than the problems in, I would say, like a lot of the online data companies, whether it's online shopping or online advertising or social media. Like it's a different problem space you're working in. I think the margin of like error in terms of lost learnings is different as well.
That's a great question. So as a data leader, for sure, because like a lot of what we talk, like people are people and they unfundamentally need like a few things or they need to be able to have a good flow state. They need to have like a bare minimum of good tools. They need to feel like they are respected in their environment and have that psychological safety. You know, they need to feel like they are connected to something that's greater than them, right?
Like I think those are usually the core things that people get motivated by. And that's true whether you're working in like tech industry or working in political campaign. I think, you know, obviously political campaign, you probably have a higher likelihood to be super connected to the mission because it's usually you've made the choice to do it because you're really motivated by the candidate or the party or the political topic.
So my like learnings and expert, my expertise, I even, it's weird saying that, but my experience managing people, I think translated back and forth between politics and tech pretty fluidly. And I think, you know, it always makes me better. I always mess things up and then I learn from them and then I get better.
I think on the data side, like it's, it is a little, I mean, there's obviously a lot of similarities. Like you need like strong data quality foundations. I think the biggest difference is just scarcity of data. So like in political campaigns, mostly the outcomes you care about are getting people to vote.
Voting is an offline behavior, whether you're in person or whether you're doing it by mail. It's also a very sporadic behavior. Maybe you're getting it once a year if you're lucky and you have a super motivated voter, which is usually not the type of voter you're trying to talk to, right? Or you're getting it once every four years or maybe not at all. So that's really hard.
Maybe someone's like, oh, but you can obviously just connect them to like all of their online behavior. And obviously, you know, magically you'll know which news sites they're reading.
But you know, it turns out like most people don't spend a lot of their time online. First of all, connecting those things is really hard. And second of all, most people don't spend their time, most people are not that politically engaged, unfortunately, you know, and usually the people that you most want to reach are the people who are least likely to be politically engaged, you know, and they are, what they're doing is buying, you know, Ugg boots and they're, you know, clicking on an ad and they're like watching a dog video.
And that gives me very little information about like how to convince them to like some candidate.
So anyway, it's I mean, it's a really challenging, super fascinating problem.
What Shopify does and the data it generates
Like there are some really smart people that are working on it. But it is, you know, in comparison, being in a place like Etsy or Meta or Shopify, it's like us, like I'm just like spoiled by data, right?
Like you get the online, you get so much data about, you know, every little thing someone's doing. And it makes, you know, creates different challenges. How do you manage all that data? How do you make sure it's high quality? How do you sift the junk from the good stuff? How do you like leverage in all the right ways for impact? And the feedback loops are much, much, much faster. But yeah, it's a little bit like, yeah, it's a different beast in terms of the type of data you're working with.
I also realized this might be a good chance. Maybe I'm coming in too late to explain a little bit about what Shopify is. I think a lot of people know Shopify, but I'm realizing we really... Shopify is one of those companies or technologies. It's kind of everywhere. Like everyone uses it. Maybe not a lot of people don't realize they're using it.
But also, also put people in the ballpark of when you say data, just roughly what, like maybe just in broad strokes. 100%. I know. So, great question. Shopify is, it's a platform for merchants or a platform for businesses. Primarily we're selling goods, you know, online or in real life. We also have a retail offering. And it has millions of merchants, you know, using the platform to reach customers and consumers.
So there are some very big brands that everyone would recognize that use Shopify, you know, like your Skims and your Alloyoga and your Allbirds. But there are millions more of these like amazing kind of like long tail merchants that might be in your local neighborhood. Like my local yoga studio is a Shopify merchant. Yeah, they have an amazing little store and you can buy from there. As well as, you know, ones that you might know for like subscriptions, things like that. So lots of small businesses and large businesses on Shopify.
And it's really the infrastructure. It's the pipes underneath. It's how stores can build a storefront but also manage their inventory. How they can email and talk to their customers. Right now there's a whole bunch of work, obviously, to make all of this agentically really easy. You can basically take the shop online.
Yeah. And it's really a play of like, hey, if you're an entrepreneur building a business, you're passionate about what you're building your business. You're probably not passionate about like writing the code to run a website. And so like how does Shopify take that off the plate so the entrepreneur can do their best work as an entrepreneur?
So the data we have is a lot of online shopping data. Right. So it's, you know, millions of buyers making purchases from merchants around the world. And it's an amazing view into online commerce. It's one of the major engines of online commerce. So, yeah, data about what people are buying is basically the very long-winded answer to your question.
Yeah. And I love your, it sounds like you do data infrastructure for the companies that is infrastructure for shopping. So it's like infrastructure all the way. It is. Absolutely. Yeah. No. So it's a lot of infrastructure. And again, this is the first place I've managed this large data infrastructure team. I did have it at the DNC. I started my career on the analytics and data science front. So it's kind of like I've migrated down that spectrum to get more and more closer to the infrastructure.
Just out of morbid curiosity, were you like Python or R or what? Python more than R. But like, I mean, neither really that well, honestly. Like, I'm just gonna be honest. I mean, I write, I mean, I would say my coding skills were never like my strongest technical skills.
I'm sure you were a contender, you know. Though I'd like to say my very first coder, it was VBA, which really dates me because, yeah. Horrifying look over here. Yeah. VBA. Yeah. A little too much VBA in my past, unfortunately. Yeah. Terrible memories.
I mean, the way I always describe Shopify to people is I think, you know, part of the reason why Amazon got so popular so fast is that people didn't like typing in their credit card information and their addresses into, you know, slightly different ways into every website. And so now, like, whenever I shop from an online merchant that uses Shopify, I'm like, I don't have to type in all my stuff. Like, it just remembers my credit card information and, you know, it has my addresses. And it just kind of makes the whole, especially like makes me feel better about like not giving all my money to Jeff Bezos, you know.
AI adoption and the current moment
Fair. But speaking of, you know, speaking of agents, I mean, Shopify was in the news last year from, you know, its founder, founder Toby, you know, published an internal memo around adoption of, you know, incorporating AI into everything that the company does. And so I think we find ourselves in this really interesting moment in January 2026, where like, I think we had, like, I feel like last year, maybe like last March was the, there was this inflection point with the release of Cloud Code and like the first coding agents.
Yeah. But something has happened in the last 60 days, which has triggered I feel like it was kind of going like this, and then like this, and now it's just vertical. It's not even a hockey stick. It's just a straight line up. And I'm really curious.
I think there's navigating overall what does it mean to incorporate AI into all the different ways that a team like yours works, but now with essentially engineers going into overdrive using coding agents to build things. Also, I can imagine that that could create a lot of chaos with a million agentically engineered side projects and essentially people wanting to reevaluate every piece of technology they use and whether they could replace it with something that's more bespoke and more specific to their particular needs. And so I feel like it's completely upended the whole, not for all things, but for certain kinds of things, like the buy versus build calculus of like, could we spend a month and let somebody loose with a fleet of agents and build something that's a better solution, but it's just for us?
Yeah, no, I mean, it was a pretty big shift when like Toby wrote that memo, it was released internally, and then it found its way outside of the Shopify. And I would say I feel very lucky that I was at Shopify at this moment in time because that mandate created a lot of, you know, I say it took away a lot of the noise that was happening at the time of like, should one or shouldn't one? Is this like a direct assault to one's craft and one's discipline? Like, you know, what does this mean for the future of work? And I kind of just said, future is here, like figure out how to use it. And like, that's actually like fun, you know, and when you shift from like the fear-based mentality of like, how is this encroaching on my job to a, this is another tool and how do I use it to its fullest potential to like do my work? I don't know, it just kind of really shifted the, I think, the feelings around it.
And I've seen such incredible work happening internally. I'm like continually humbled by how like smart and creative people I work with are. I was, one thing that I was incredibly proud of, like our, I think Anthropic announced like the model context protocol for MCPs in like November of 24. And my team had built one, when I say my team, but I was like, I had nothing to do with it, which is like the smart team did it. By like February, we had one for our data warehouse, which is like core. So very, very quickly, people could start just like doing all of the fun analysis. And I think that was again, like a good six months before a lot of other companies were doing something like that. And if a lot of companies still probably don't have that.
And so that's an example where I think it's creating a lot of like flay. And like, you know, and I know it's like a scary thing and it feels like uncertain and we don't know where it's going, but it's also like fun. And I think that it's kind of creating a lot of opportunities to just try different things out. And that is sometimes rare. And it's, I mean, it's, again, it's part of the like, the cool part of work is when you get to try something, learn something new.
So I'm still grappling a little bit as a leader of a large team of how to like, watch all the fun and tinkering, but also make sure we're creating scale and leverage because I want to make sure everyone can kind of be along for this journey. And like, you know, there's a lot of variance right now. So I'm, that's one of the things I think about the loss the most. I don't have a great, easy answer right now of how I'm like walking that line. I am thinking a lot about how to converge on like one default wave, leveraging it. But yeah, it's a hard question. I'm having a lot of fun with, with, with agentic coding. Like I was just watching, you know, Peter, Peter Steinberger is the creator of, you know, the now viral Claude bot. Yeah. You know, he gave a talk late last year. The title was, you can just do things. And so, you know, it's kind of like, and I wrote a blog post that's called why not? And that was kind of like my new motto of like, Oh, you want to build a thing? You know, why not? Like, just try, you know, build something. It doesn't work. It's not good. You can just throw it away and code. Code now has, you know, much less value, you know, used to be that like code artifacts would be like, Oh, this was the product of human labor. And like, we can attach a cost to this. And like, you know, it's always funny. And like the, the code counting pools, like SLOC and CLOCK and SCC, they'd be like, Oh, this code base would have cost $3 million and three years to build and be like, I built it today with my agent.
The title was, you can just do things. And so, you know, it's kind of like, and I wrote a blog post that's called why not? And that was kind of like my new motto of like, Oh, you want to build a thing? You know, why not? Like, just try, you know, build something.
And so it's, I think part of what's fun is just the, you know, not knowing what's the best, you know, like, what's what's the right and proper, proper way to do things and encouraging like that experimentation and creativity. And I feel like for me, like the almost like, for me, it's like it gives me what's happening right now. It gives me the same feeling that I got when I started to do Python, which was, you know, like in the late 2000s, where, yeah, it's like, I can just, I can just write code and do things. And so now I feel like 15, you know, almost 20 years later, it's almost like that same feeling again of like, you know, I can just, we can just do things. And, but also, like I can imagine in a, you know, in a company setting that that that, you know, trying to channel that enthusiasm and that like zealotry towards productive, you know, productive ends is going to be like, you know, in a sense, like, yeah, I guess probably, you know, maybe the challenge will or the strategy is to, you know, let engineers, you know, enjoy their enjoy their tokens and yeah, try to try to understand, well, you know, that was a good idea. Maybe that wasn't a good idea. But we ran a lot of experiments and some of them worked and, you know, probably maybe three or six months from now, some of the like, fervor that's going around, like the zealotry will, will dissipate and people be like, okay, this is the best way to use agents to build stuff. And we won't have, you know, Steve Yagi rolling out, you know, wacky, you know, orchestrators with pole cats and deacons and whatnot. But it's, yeah, I think the uncertainty is part of what's what makes it so fun.
Yeah, no, I mean, I mean, I also kind of feel like the most of the best things in life, it's like they get more interesting, the more you engage with them. And I think that's true of like AI and LLMs right now. Like it's the first couple of times you do something with them, you're like, oh, this is like magical. But then it's like, the more you use, the more, almost the more magical they feel. And like that part of it is actually really exciting because it's like, you kind of want to keep going down the rabbit holes and pulling those threads, which again, to your point can be like a little bit distracting if you're like, hey, we also need to all do our jobs and, you know, do impactful work.
Exactly. Exactly. But I think those two things can go hand in hand. Like, I think that there's ways to like, you know, I think most of what I see because also Shopify is a pretty fast paced company with a pretty like, you know, rigorous set of milestones to hit. People are mostly doing all the creativity for the purposes of like impactful work. And so that right now, I don't worry too much about it being like a bunch of side quests. I think that they're all pretty aligned. But I do think it's uncertain. I also think it, you know, if six months ago we like locked it and we're like, all right, now this is the way to like AI, we would have made a bunch of bad decisions. So I think it is probably is premature to be like, hey, here's the right way to do it. And I think also a lot of people are still just figuring it out. And so I do also think like demystifying the idea that there is a right way to do anything right now is important because no one knows. Like we're all just stumbling around. And, you know, the next cloud bot's around the corner, you know, it's like you never know what's going to emerge like in two months or four months that's going to change how we thought about things a year ago.
Uncertainty and the history of technology
Yeah, it's interesting to hear. I think because it reminds me like Charlie Marshall also mentioned in another interview, like this sense of uncertainty, like as a leader, the immense sense of uncertainty with AI and I think where you mentioned like who knows what will happen like three, six months from now, like how do you, yeah, how would you like quantify, not quantify, but like if you had to describe how much uncertainty is there, you know, is this like a level of uncertainty that's like you've seen before or does it feel like kind of one of one?
That's a great question. I mean, I do think this is for me the most uncertainty I've felt about how a technology might evolve. I was also like, I emerged like from college like post-internet. Like I don't know, people, I think some people compare it to like early internet days, which was like, I didn't quite experience that. 1995. Yeah, I was still a wee one then. So, but I think for my, for where I am in my career, I think this is the moment of the most uncertainty. What I will say though, a fun fact is I studied the history of science and technology and there certainly are moments throughout history where there was immense periods of uncertainty about emerging technologies. So I don't think that this is like one for one in the history of how humans have used tools. I think it's a very big one though and certainly like the biggest in my lifetime so far. But it's a pretty common pattern actually. When like a new technology emerges, whether it's like the car or like the fax machine or whatever, there's like a period of misuse of the product usually, right? Like, I mean, there's classic example of email, but where technology like is changed by the users. And actually, if I can quickly do a quick detour into history of technology. I mean, we can make this whole thing about history of technology. That's a crazy bone of credential to drop. You're like, I dabbled in a little bit of history of science technology. That was one of my career side quests.
I think like the sort of like simple narrative of the emergence of technologies is usually like the founder myth of like, hey, this one person invented X and they knew exactly how it should be and how it was used. And it's like, they get all the credit for inventing like the thing. But in many, many, many cases, the technologies get like fundamentally altered by the people who use them. And like that usage of those things like changes the trajectory of them in like unexpected ways. And that it's a very much like a co-evolution once technology has emerged between the technical practitioners, the users of the technology, and often like regulation. So like that's also not unusual where you get people like how do governing bodies start to play into how the technology can or can't be used. So we're at that moment right now where like all of those things are a little bit up in the air. And I think what's very cool is AI is something that is so clearly not like a founder myth. It's not like there's one person who's like, and yes, now you all can benefit from my great tool. Like it is. And we're going to see all of these like little, you know, mini founder moments of people being like, I just developed that or this or that. But it's such a collaborative moment, really, when you think about it between like all the way different ways people are like sharing what they're doing and being able to kind of inspire each other. So I think that part of it's very exciting to see the kind of co-evolution of this technology. And it's so inspiring, I think. And yeah, for real, like fun to hear from that perspective, like versus a force that's like threatens to crush you, the like co-designing kind of like exploration. And it's a really interesting point to that.
And I think what's very cool is AI is something that is so clearly not like a founder myth. Like it is. But it's such a collaborative moment, really, when you think about it between like all the way different ways people are like sharing what they're doing and being able to kind of inspire each other.
Yeah, it is interesting. So many people are sharing like what they're building and stuff right now that it's, I don't, it's a really helpful perspective, I think, to hear. And it is. It is. There's a lot that is scary. So I don't want to gloss over that and be like, oh, it's everything's just like all in the up and up. Like, obviously, there's massive like existential questions at play. But I think purely thinking about it in the bubble of how do we use this to do better data work? Like, I think it's a lot of, there's a lot of fun. I mean, I think in terms of, you know, being able to create, like in the past, you would say, oh, well, we can't justify staffing a team to build like internal tools to help with, like, I don't know, certain types of like, you know, very particular like platform observability or like, you know, essentially internal systems which are used for development. And so I think now like that, that that cost equations like you can, you know, there's there's a tool which can make like these 10 people more productive. And, you know, in the past, I'd be like, well, we can't can't have three engineers spend six months building a tool that only makes these 10 people more productive, like they can just get by the way that they way that they are. But now, you know, it is possible for small, you know, there might only be a small audience of people that benefit from creating some, some new tool, but if it saves them an hour a day, they spend they might spend a day building the tool. But then if it saves them an hour a day going forward, like that's that's material, material savings that can direct that direct that that saving time savings toward something else that could help the company.
Metric investigation and the semantic layer
I realize I've been speaking a lot in like, generality. So actually on that topic, if it's useful to give like one more concrete example of like how I'm seeing this play out right now. Like a very common data science problem, analytics problem is why did that metric go up or down? Right? It's like, what happened? And usually the answers to that are like, one seasonality not being properly controlled for to a data pipeline issue. Three, there's actually some meaningful movement here that we need to understand. I think that there's so many ways that automation slash AI can really help like run through the set of like initial hypotheses and jumpstart, jumpstart the investigation into like further into the interesting problem space, right? And in that sense, like there is a little bit of a playbook most data scientists use of like, OK, here's the ways I'm going to go about like investigating, like, can you get those reps in like kind of faster, kind of like in a less costly way from yourself, you know, using some automation, using some AI and allow you to get faster to the point of like interesting work. And I think that that part is like, that's really cool. We're doing a lot of work right now to try to put better best practices in place and governance in place around defining metrics. It's like basically your good old semantic layer stuff, but defining metrics, segmentations, reusable segmentations or dimensions that you care about. It's like making sure everyone's cutting by the same things, building better benchmarking tools. So how do you use profit in the best way or whatever you're going to use for a time series? And then how do you build alerting and automation on top of that? Like, so that's kind of a good example of like, there's kind of interconnected set of things from the like, how do you define the metric in code? And then how do you like get to understanding how that metric is moving in the most meaningful way as possible and like using good practices across all of it and AI where it helps get it to us a better outcome faster.
Yeah, it's interesting to hear. So you're saying like there's when a metric changes, like a data scientist kind of has like a list they can go through to kind of pull that thread. And as it turns out, like if it's a needle in a haystack, like AI can be pretty good at kind of like at least taking a pass at it. At least taking a pass at it. Like, you know, and especially like usually it's gonna be like, all right. Okay. Is it like moving in all regions or just a couple regions? So we're gonna look at that. Okay. Oh, it looks like it's mostly moving in the U.S. Okay. Like, is it based on the age of the cohort or whatever? Okay. We're gonna like look at that. Okay. It looks like it's actually primarily coming from this segment or no, it's widespread. Like those are the type of questions that like, you know, you have to have the right model in place, but going through those permutations, I think you can use automation to do so that it's not like hand, like again, there's a lot of exploratory analysis is so fun and important. You don't want to automate all of that away, but it's like, can you handle, can you like help get your, it's like having a metal detector. Can you like get like, okay, it's over in this area. We want to start over here. And that's where you kind of like to jumpstart so that you're not just doing the like macro map. And a lot of times people like, they don't think to cut by like, I don't know, X, Y, Z dimension. And so I'm gonna look at it. And that's actually where the interesting nugget was. And they spend all their time over here. So again, data modeling is still the most important problem in that though, because you have to have, you can't do any of that fun automated analysis if you don't have the data modeled properly.
The semantic layer challenge
So you mentioned the semantic layer and that's, that's been a hot topic at various times. I don't know. We can totally skip this, but I almost wonder if it's worth just saying briefly what a semantic layer is to you just to kind of set the stage for who might be listening. Also that we probably shouldn't use this bit, but like I would say semantic layer decisions have been my greatest failure as a leader because yeah, there's right. Anyway. Yes. What is semantic layer? I mean, basically it's a way to translate between a defined metric or like defined business logic and the model data that you have. And it's reusable across like the company, right? So that you can always reference the same thing. Some like Looker has a semantic layer that's like built into it, LookML, right? And that is like the magic sauce that you're stitching together the data sets and then the sort of like business units on the other side. Many companies use different versions. It's circling around the semantic layer and what the right tooling choices for semantic layer has been something I've been grappling with internally. And I had a lot of conflicting input from my team and I had a really hard time navigating it. So it's something that we were re-interrogate.
Also that we probably shouldn't use this bit, but like I would say semantic layer decisions have been my greatest failure as a leader.
Honestly, it's so helpful to hear. I feel like that, like if you're grappling with it, I'm sure everybody else in the world is grappling with it too, you know, like. Yeah. And it's also a good example where just sometimes, I mean, this is one of the challenges. Like I have a lot of different stake team. I consider my team a stakeholder, like different components of my team. I have different needs out of what they want out of it. I have a team that just does what we call executive insights. They're looking at cross company trends. And then I have other teams that are deeply embedded in a product area. So they're just working specifically on our storefronts or on our checkout or our shop app. And their use cases might be different and people get attached to what they've been doing. They're like, don't give me some new thing. I have my semantic layer I like. I use it over here. And so that's where like unification I think is really important of like getting everyone on the same thing. But it also comes at the cost of people migrating, which is never fun. And so that's where like some of my job is to have to make the hard decisions and just be like, yep, we're all doing this. And yes, I'm sorry you have to do it. And when I don't have clarity on what the right tooling choice is, it's the hardest because then I'm not confident that the juice is worth the squeeze.
Yeah. Is there any way you can describe like the difference between some of these tools? Like what? Like the difference between different semantic layers? Yeah. Like what's the variety of flavors you can? Yeah. I mean, some of it is like how homegrown you are versus not. So like there are versions that we could use that were more dbt like reliant versus not was one example. I mean, look at like whether to go on and Looker was another decision we had to make. So like there was what is like how much how how much dependency you want on a third party tool and how much investment you want to make in your own your own tool. We were also like we were hitting a limitation with some BigQuery. Some BigQuery like there's news of how BigQuery was handling like changes in datasets over time that was like was like we were running afoul. So like there was like we are hitting technical limitations. We were kind of debating the future of our commitment to Looker, which we are not really committed to. And we were also debating whether or not we wanted to get all these people who were on separate systems onto the same one. So like those were and we have like one internal tool that uses a different one. But it's not like itself. It's a very small amount of code. It's not like it's a major. It's not like it's a huge investment. It's just more. Yeah, the the users had different perspectives on what their most important priorities were.
Yeah, I mean, Looker is. Yeah, I mean, I would say Looker is probably the most successful semantic layer company. Well, essentially, it's a semantic layer company that that, you know, looks like a business intelligence company, kind of business intelligence and semantic layer in one. And the creator, you know, creator of LookML, Lloyd Tabb, has a new semantic layer called Malloy that I've been Malloy that I've been following, you know, quite, quite actively. But then there's, you know, there's other companies like cube and other ones that people are, people are using and probably others that I'm not familiar with, because I'm not, I'm not doing, you know, enterprise data modeling. Yeah, very much. But, but it seems like it's important to have the team invested in whatever, you know, whatever you're using. And that I can imagine there's also like the need to have like good, you know, telemetry about like, what's actually being used. And so that, you know, because you could have like blind spots where like, this part of, you know, the semantic layers being maintained really well. And we have good confidence that like the queries that are being written based on it are the right ones. And, and the, you know, the analytics are correct. But then, you know, there might be like dark corners where it just hasn't received the attention, or maybe it's like, drifted, because there's underlying changes. And you know, the data warehouse or things like that. So and we've invested a lot in data quality metrics for the data warehouse. And so we, the, the analytics engineering team I mentioned earlier is responsible for this. We have a monorepo, basically, of our data warehouse. And so Shopify is also very big on monorepos. So it's a kind of monolithic development environment.
And so that team is, is really focused on the ergonomics of working in the data warehouse, and the quality of the data in the data warehouse. And actually, you have to like meet quality standards to be in the data warehouse. And there we've, we've had, we have a big effort around like measuring documentation, completeness, measuring, obviously, like freshness and timeliness, measuring usage, and kind of a bunch of tests as well about like completeness of data. So that's, it's kind of again, that's to me is like, that's age old problem that like, it's really important. You can't like, you know, you always have to be investing in.
I feel like we've talked about so much from like, data cultures to you kindly laid out the whole stack for us, which I feel like is a Herculean task. And I think attempting to tackle a semantic layer is a special form of health. So I really appreciate it. Yeah, I didn't expect us to talk about the semantic layer. It's a tough, but I do think like, just wouldn't, wouldn't, wouldn't be if we can't mention the modern data stack and not talk about semantic layers. So hot. Yeah. So hot right now.
Sneakers, Groundhog Day, and unwinding
I do think this conversation would be complete, though, with the fact that you did mention your last podcast was to talk about the movie Sneakers. You came on as a sneakers expert. Is that like on the side, you do a lot of data stuff, but also you also consult it sounds like on podcasts. That's right. I have a very specific movie knowledge of one movie from the 90s. You know, I do love the movie Sneakers. I am not an expert, but I am, you know, a lover and champion. If anyone has not watched it, it's my people often ask me like, what advice would you give to someone earlier to date a career? And I'm like, watch the movie sneakers. It's incredible. Yeah. Incredible. I mean, we'll teach you nothing about data, but it's just a good time. And like, it'll give you a good laugh. It'll make your day better. Is that? Yeah, it's like, it's, you know, I'm getting good warm feelings. So like admire, you know, great actors. Is that, do you tell people if they're getting into data work, they should watch sneakers because that's the right answer? Because that's your answer to everything. That's the data scientist in me, you know, I'm like, what's this model doing?
It's a good movie. I only, I only saw it in the last 12 months. And so I made it until now. And it's still, it holds up. I mean, it's, it's definitely got some 1990s charm, but for me, like the 1990s movie that I've seen the most is Groundhog Day. And so like, I have very, like, very, very good knowledge of that movie. But yeah, I think that I would, I co endorse sneakers. I haven't seen it. Groundhog Day is kind of like a data science movie too, because it's like what happens with repeated. Yeah, no, apparently I didn't realize that there's a whole like, there's a whole like, kind of, you know, cult following of Groundhog Day and a lot of like alternate theories of like what the movie is really about and different things. And I was like, it's, you know, I think it's like a nice, it's like a really nice romantic comedy. Like who doesn't love like the cast is great, you know? Yeah. I love those corners of the internet, though, like when they're positive, you know, when it's like, oh, this is we're gonna have fun with like, going as deep as we can into the mythology of like this particular movie. Like that's, that's like people at their best. I do feel like it's closely related to the activity of trying to connect cinematic universes. Like, yeah, what if Groundhog Day and sneakers occurred in the same? They were released very close together. Right. Coincidence. Oh, great movie taste. So you need to get it on the night. I gotta get into sneakers. Yeah. And I feel like now that we know you're a cool person. Maybe just wind up. Yeah. How do you like to unwind? What's your, when you're not watching sneakers? I flip on the podcast with me talking about sneakers.
No, what am I doing online? Well, I mentioned at the top, but I have two little kids. So I wouldn't say that's unwinding, but I fulfilling, fulfilling. And my actually my recent unlock with them, my youngest just turned three is that I've introduced them to Mario Kart. Oh, my gosh. I'm by no mean, like an excellent Mario Kart, you know, I'm an enthusiast, but not a expert. But it's so fun to play with them. They when I say play, I mean, they, the three year old just like hold the remote and it like auto drive. My five year old tries. As you're blasting them with shells. Well, weirdly, she usually comes in like eight. And like my sweet five year old who's trying really hard, always. But it's that's been like very fun to introduce them to like a little nostalgic throwback. And also, yeah, good to, you know, just have some silly fun. It's underrated, just like fun. Yeah, that's so sweet.
Yeah, I really appreciate you coming on. And I feel like such. Yeah, it's so inspiring to see how you've like woven between different jobs and and balance like impact and industry work. Yeah, really appreciate you weighing in on the modern data stack and AI. And I think helped me wrap my mind around what it means to lead a group of 400 people, which I'll still have to spend some time marinating on. Yeah, I'm still marinating on it. Yeah, we're all three years in. So, you know, we're all still figuring this out. It's that's the biggest thing is that it's all it's always a little bit haphazard how we like make our way through these choices. So, yeah, really appreciate having you. Thank you. This is fun. This is really fun.
The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio Agi. For more episodes, visit thetestset.co or find us on your favorite podcast platform.