image: thumbnail.jpg
Brad Zielke @ Target | Data Science Hangout
Brad Zielke is an Operations Data Science leader focused on unlocking the retail experience.
________________________
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back to the Data Science Hangout. I'm Rachel, I lead Customer Marketing at Posit, and I'm so excited to have you joining us today. If it's your first time joining us, the Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. And we get together here every Thursday at the same time, same place. If it is your first Hangout, let us know in the chat because we would love to welcome you in for anybody joining the first time. We are all dedicated to keeping this a friendly and welcoming space for everyone, so we love to hear from you no matter your years of experience, titles, industry, or languages that you work in. It's also totally okay if you just want to listen in here too. Maybe you're on your lunch break or you're out walking your dog, but also awesome to be a part of the party that will happen in the Zoom chat. So you may meet a few people there in the Zoom chat, see a lot of great resources shared as well.
But there's three ways that you can jump in and ask questions or provide your own perspective on topics. So you can raise your hand on Zoom and I'll call on you to jump in. You can put questions in the Zoom chat and just put a little star next to it if it's something you want me to read out loud. And then third, we have a Slido link where you can ask questions anonymously. We do also, I sometimes forget to say this, but we do also have a LinkedIn group for the Hangout, if anybody wants to join there. It's not great for discussions, but it's awesome for being able to connect with everybody and more easily find each other. Quick note though, if you are watching this as a recording on YouTube in the future and you want to join us live, the link to add it to your calendar is going to be in the details below. And as always, there's no rule that you have to stay on the whole time or talk, come and go as it fits your own schedule. But with that, I am so excited to be joined by my co-host today, Brad Zelke, Senior Director of Data Sciences at Target. And Brad, I'd love to have you kick us off here by introducing yourself and sharing a little bit about your role and also something you like to do outside of work.
Perfect. Thanks, Rachel, for having me. Nice to virtually meet everybody. Really quick, so I'm a Senior Director of Data Science at Target. I work in the operations space, so my team supports various decision modeling capabilities across everything from vendors and our supply chain and transportation providers all the way through to what happens in our stores and also how we operate in our properties world. So when we talk about properties, we talk about construction, we talk about energy optimization, work like that. And so it's really fun space across Target because we actually touch everything that physically happens. So I have a fairly large global team that really makes it all happen.
A little bit more about me, I've been with Target 20 years, so I have one job before Target and then I've been here ever since. And I'll get into a little bit of this as we talk more, but I think the variety of work in data science and retail is just fascinating. So across my 20 years, I've had opportunities to work in cybersecurity, work in supply chain, obviously stores. I've been able to work with HR analytics and other things, and it's just a really neat place where you can kind of cross domains in the industry as well. And then what I enjoy outside of work, so I have four kids. Four kids keeps me busy, but I love coaching sports and anything athletic and outdoor. So I spend my time running. I'll run my sixth marathon this fall. My sons are swimmers and hockey players, so my winters are packed either in a pool or on the ice. And so that's what I enjoy doing outside of work.
I love it. That's awesome. I was going to ask what sport you coach. Is it hockey? Hockey, yeah.
Evolution of data science at Target
Well, thank you so much for joining us here today, and I get to start us off with some questions as we wait for them all to come in from everybody here. But I was curious, Brad, being at Target for over 20 years, could you share a bit about the evolution that you've seen of data science in the retail space?
Yeah, absolutely. So Target probably, I've been around a long time. So I started at Target doing statistical modeling in distribution centers. And what's fascinating is when I started, we only had 16 VCs in the country. Right now, we have over 60. And so you think about those localized models and solutions we were building. I've been part of seeing that grow and rapidly expand, too. At one point, we centralized that for the company, then we've extended it. And then over the last 10 years, this is really my third stint in and outside of data sciences at Target. So at one point, BI and data analytics were all part of data sciences. And as we brought that together, we decided to build data products for Target. We started to formalize measurements and metrics across the organization. And then data sciences really grew rapidly. And so we talked a lot more about decision models. So anything from recommendation systems to search to optimization and algorithm development that we do in operations today.
And so I've seen this huge transformation into being very supportive and augmented to the organization, to becoming more and more data science-led and forward, especially in these spaces where I think traditionally it would have been just statistical models or more of our objective decisioning and now becoming a lot more augmented in terms of leading the organization and new capabilities as well. So it's been a really fun journey that we continue to build on. Gen AI, obviously, is all the buzz right now. And we're dabbling in that. And it's just fun to see what else is possible.
And a quick follow-up question from your intro, which marathon are you running next? I'm running Twin Cities Marathon for the fifth time. And then I'm looking at doing Grandma's now that our winter has been so warm. We've been able to get outside and get some more miles in earlier in the season. So I might run two this year.
Data science and operations use cases
So just so everybody knows here, when we say data science and operations, could you share just an example of maybe a use case that you're working on now or what that really means?
Yeah. So as I alluded to, our teams work across all different parts of our operations, from supply chain to stores to properties. We work heavily in the optimization, OR, operations research space. So we build simulations or digital twins of what we're doing across our different businesses. We do a lot of forecasting work where we're helping outline what networks should expect or how things will be moving or how to staff a store. And so our team works across all of these different operational teams or trying to execute work in the field.
And so right now, I think one of the things I'm most excited about in our space is how we're using the horizontal capabilities of simulation optimization to look at the inputs and outputs as we derive a network. As we derive a network of capabilities from what we buy and how we move it all the way to how that will end up in a target building. And so we can start to look at what we're doing internationally, how that lands domestically in our ports or our decons, and how that ends up moving into our stores ultimately. And so we have models and algorithms that are starting to outline and connect different decision variables that happen upstream into those decisions that we're making downstream as well.
So when you go into a target, are you thinking about all those things as you're shopping? Yes, absolutely. Often I'm either annoyed where we don't have something that I want because that means something else happened wrong upstream or I'm overzealous about something that looks great because all the work that ended up making that reality.
Something I meant to add in the beginning, I wanted to just remind everybody, if anyone's hiring for roles, feel free to share that in the chat as well. I know there's some people who are looking that are joining as well. So it's not spammy at all to share those roles in the chat too.
But I guess, Brad, I'll ask you as well. Are you currently hiring for any roles at Target? Yeah, we're always hiring. So I'll make a plug where we're building out some new search teams. We're looking at work and our recommendation systems and personalization capabilities. And then even across MySpace and kind of the operations side of the team, where we're always looking for talent to augment and build out some of our teams here locally in Minneapolis and then also in our Sunnyvale office. And some people maybe don't know this, but Target does offer remote and hybrid work too where we have teams that travel in and out across the country. So feel free to reach out to me afterwards if you're interested.
Awesome. Is LinkedIn the best place to connect with you? Yeah, that's a great way to reach out. Okay, perfect.
The Target pregnancy prediction story
Curtis, I see you have a question here in the chat or maybe you're starting a conversation here in the chat. Do you want to jump in and ask that? Anytime someone mentions Target and data science, the anecdotal story about how, I think it was the marketing data science team had... Of course. I just wanted... Basically, anytime anyone from Target shows up, I always ask them to confirm the story. So maybe if you wouldn't mind kind of recapping it and sharing. There's a few people who signed in their classes.
Oh my gosh, it's like the atypical academic case study, right? Of like machine learning and machine learning of like machine learning and data science gone wrong. So those of you guys who maybe haven't seen the article, obviously, or maybe you have and you just want to refresh here. But Target started sending recommendations ads to a family. And the father found out through some of those ads and details that his daughter was pregnant, which then turned out that Target knew more about the situation and what was being purchased based on that behavior that provided us details and insights of how to use that and offer other incentives, which could be pretty intrusive.
But I think it was also very interesting and highlighted the power of data. Now, how Target handled that and all of that, we, I think, had to disclose some more information, I think led to other processes and good governance that we're working through. But it's a great case study, a good example of the power of it, but also how to be a good steward of data science as well.
But it's a great case study, a good example of the power of it, but also how to be a good steward of data science as well.
That's the one you're alluding to, right, Curtis? Exactly, yeah. That's a very true story that I remember very dearly. It must be hard when something is used as a case study in a lot of classes, but then becomes something that is always asked about too. Yes. But there's so much more to data science than that, right? I think that was obviously something that made the news, but tons of good things with it as well.
Engaging business partners and building understanding
I see, Rick, you just put a question in the chat. You want to jump in here? Oh, hi, Brett. Thank you very much. Rick Hubbard here. I'm a retired practitioner and a university professor, but I'm very intrigued by your remarks. The question I have is your DS function sounds to be quite sophisticated, quite capable. Sounds like you have a breadth and a depth of competencies. I'm wondering to what degree do your business customers, the other lines of business that you interact with, to what degree do they understand DS and the work products your team produces, methods and practices you undertake? And in addition to that, what accounts for the business side's degree of acumen in understanding who you guys are and what you do and how you do it?
Yeah, I'll start with the first question. Our customers don't always understand what we're doing. But I think what I emphasize with my team and some of my teams on this call is we have a deep empathy for our customers, our partners, and the decisions they're making. And that's always our starting point. I think that gives us a way to interact, teach, educate, and bring them into and along with the solutions we might share with them.
I'll give you an example though. So I just took my whole leadership team out to our office in California. And then we all went and did an immersion experience where we went and actually walked physical buildings. We met with team members on the DC floors. We met with store service team members. And really we're highlighting and showing where our work intersected with what they see and do every day. And that hands-on connection allowed us to get feedback in terms of what they see from our work. Also allowed us to build, I think, a deeper understanding of why we see disconnects often from our work. And then just enabled us to even further educate people in the field who maybe are like, why is this happening? Some of the nuances of what we're doing in our models that end up leading to that.
And so we end up augmenting our stores with different decisioning models that change the tasks or the work that they have. And when that changes, we can tend to see questions of like, what is driving this? This doesn't seem normal. And so then we can kind of loop back with them and have more of a high touch education or connection with them as well, which I think goes a super long way. And so we spend, I think, as a leadership team, the majority of our job actually engaging,
educating, and partnering with them to teach, train, and then drive kind of the focus on this work versus delivering it. And then we have really, really talented teams who end up building it, interacting with them as we deploy it and trying to capture as much feedback as we can.
So we have a lot of success criteria in the operations space specifically, where we not only just deploy and then test and learn virtually, but we actually go and audit and look at our physical impacts as well, which I think is really, really unique.
Brad, one of the reasons I'm asking is years ago, to show you how old I am, but you can tell from my white hair, I helped implement what was called a information center. It was considered to be revolutionary. My gosh, how old is that?
But one of the things we did that I, so we implemented this information center and it was in largely a financial institution, broadly just geographically distributed financial institution, and early results were desultory. So I thought over and said, what's going on here? And I realized, I went out and talked to some of our prospective business customers, partners that we would support in the operation side.
They didn't know how to engage with us. They knew we existed. They knew we could answer certain problems and gather data and do forecasts and all that jazz, but they didn't actually know how to engage with us. So I did a little, we put together a little course. I had my folks put together a little course on kind of, I don't remember the title of it, but it was sort of like, this is how to use an information center.
And our utilization grew exponentially once folks knew how to ask us to ask a question, if that makes sense. To what degree do you have in your outreach program to the various lines of business, operations, marketing, HR, all the functions that you mentioned a moment ago, anything like that, you know, to help them use you better.
That's my, that was my objective. I wanted to help my customers use us better. And to do that, one of the things I had to do is to give them a tool so they knew how to help, how to come to us. Your thoughts?
No, I have a lot of thoughts on that. The thing I get excited about that, right, is that direct connection you pursued in doing that. I think that unlocks a lot if you can train and teach and create the patterns that actually work for them versus, you know, feed them what we think will work for them.
I'd say on our side, there's two things that stick out in our model that maybe are a little unique. One, we treat all of our models and science as products. So just as you would an application and treating it as a product and how what that user experience looks like and how you would build a roadmap around features and value and all of that, we do the same thing with our models. We want our users to have, you know, some product accountability in terms of like, what is their experience? What's the outcomes? How do we measure that? What does that look like? What's the efficacy of a model or what are we doing with it?
One, we treat all of our models and science as products.
And I think that that allows us to also give them more support as they engage with it and understand it and also helps our scientists build a bridge into like, hey, I want to go see this thing or let me go interview this user or make a direct connection. You have it both ways.
So I think one, having kind of product mindset and product connections and governing and managing this stuff takes more ownership in both of the outcomes as well as what you're building. So that's one.
I'd say the other thing that we spend a lot of time on is that training. And so when we do the model, we spend a lot of time in training what the changes will look like in their systems or in the output of this.
On my screen is Alfredo. He's on the team, but he's many times, I'm sure educated our stores partners on like, hey, this model will change what you see here. We will deliver not just the output of the model, but the reports and the views and how they compare it and use it to all their other processes. And that's the way we can get through change management and adoption a lot easier than we have in the past where it's like, just give them the report and let them self-serve. I'd say it's much more high touch in making sure that they understand it and see those components come together.
Oh, commendations, Fred. Sounds like you guys have a really good handle on it. The one thing I really liked from your remark or really stood out. I just want to make sure I jumped to some of the other. Yeah, it was the feedback loop. So commendations and thank you, Rachel. I yield the floor. No, thank you for the great questions. I just want to make sure I get to some here.
But I really do love hearing about the immersion experience. And I remember when we were talking before, it was when you were planning that trip out to California. I see Alan had a follow-up question on that. Alan, you want to jump in?
Yeah, thanks. Hi, Brad. I'm really, really interested. I'm curious about how you sort of assess things like physical impact. You referred to that. Like, it sounds like, you know, seeing behavior changes in stores or in customers or maybe in employees. Really curious about how you measure that. It seems like a really interesting measurement question. And also, it seems like fieldwork, which seems really fascinating to do in that environment. So I wonder if there's anything you can sort of elaborate on there about how you turn physical impact into data that you can then continue to work with.
Measuring physical impact and inventory corrections
Yes, so the feedback loop we often look at, right, as a combination of all the virtual records and events and the features that we train our model on and then score and then end up giving us output. But we also look at, like, what are the business outcomes and measurements that subsequently follow? We want to have effects and outcomes. And that's how it turns into the value for Target.
And so an example I'll talk about a little bit I can share. It's actually published in our Target blog, tech blog, if you want to go look at it. But we do something called inventory corrections in our stores. And so we use models that help us predict when something virtually is not there, even though physically we think it is there. And as we look at those events in our stores, what we can see is different things subsequently happen.
So we'll take a virtual record, we'll correct it. And then often tasks and workload are generated from that. And so then we'll see, did the team members go get the thing from the back room? Or did they find it? Or did subsequent other decisions in our systems change that then maybe brought that item to the store? And so we look at these physical connections and what that workload is, what the effects are.
Sometimes we make mistakes. And so we'll recommend something that's not there that actually is there. And then the team member will go actually do something with it. And then it doesn't fit on the floor, which then means they have rework to do. And that's a cost to target. And it's a really frustrating experience for the store team member. And so looking at not just the, you know, the precision of the model and the output and what value we're driving from that, but then the subsequent impacts of that has allowed us to connect back and also build trust because we care more about the whole business problem than just even making, you know, those nuanced decisions better across the whole value stream. So that's a real example.
Just to give you, you know, some scale on that, you know, we do that hundreds of millions of times a week, which, you know, every team member then has to touch those subsequent impacts. And it's fascinating to see this interaction and the costs and the interplay between the physical and virtual worlds. That's the really, really like that at scale it's really interesting and impressive. Thank you. Thanks.
Selling bad news with data
I see there's a few questions that came in about like sharing results and reports with users. I don't know if Jeb, you want to, oh, I see a star here so I can read it. But the question was, how do you sell bad news? We make data products with sound bases and sometimes it surfaces issues that might indicate an issue or concern more often than I want to admit. There's pushback on the model rather than a look at the issue.
A real life example of that is highlighted often in simulation. So we'll use and work across business teams. And recently, some of you guys probably know Target rolled out Starbucks as part of your drive up order. If you shop at Target, it's an amazing benefit because you can get your Starbucks in your car while you're also waiting for your groceries or anything else. I'm a big fan.
But as we were looking at and modeling out, actually simulating that for stores and the work there, what we found is the data told a different story than what our subject matter experts at headquarters expected to happen when team members or payroll was being used in stores. And I, you know, that wasn't a great story to be like, sorry, I know your process is defined this way and you expect all of this to happen in this way, but you're wrong.
But I think the reality is we met them there. We showed them the physical. We actually met with them in stores. We walked through the virtual records, the physical actions and activities. We saw these things changing and evolving. And what happened is we drew alignment between the physical and the virtual and highlighted those, I'd say perceptions that were incorrect and allowed data to kind of work and tell the story. And so, yes, it was bad news. But I think at the end, because we were all aligned and doing that work together, we were able to deliver it with some empathy and connection, which ultimately led to us advancing the overall solution and simulation for our stores, which helped them transform other decisions. And so they saw it not just as a serving bad news, but being a partner and then unlocking other capabilities that advanced it further. That's a real example.
we were able to deliver it with some empathy and connection, which ultimately led to us advancing the overall solution and simulation for our stores, which helped them transform other decisions.
Digital and physical experience
Thank you. I am thinking, like, as we're talking about watching people in the physical stores, like, how does that translate over to the website to or like knowing the process people are going through to find things?
Yeah, we, I think, have a great desire to be more sophisticated of blending, like our digital experience with everything else in the physical operations. If you have read in the news the last couple of quarters, our digital sales and things have taken a little bit of a hit.
I'd say overall, though, Target's been well-positioned to try to be a great place to shop and meet our guests with the demands they have. We've, you know, through the pandemic, opened up drive up, pick up and all their services. But that doesn't always fit within the same guest experience as you have if you were just to shop one of our stores. You always can see the buzz of somebody having a lot of fun at a Target store. But we don't often get the same experience digitally.
And so it's a huge effort for us to start to sync up on these things, you know, make search better, make recommendations clearer, you know, compete on some of those digital experience things that, you know, you can browse and shop in a store that Target's known for that we haven't completely been able to do. But I think we're on that journey. And, you know, our services and operations in the space that I know, we are truly trying to sync that up as much as possible to make that as easy as possible. So the dynamics of you shopping with ease and checkout with ease should be as easily as you are as easy as your store walking through a checkout line as it can be digitally. And I think those are just nuances of try to try to blend those two experiences together.
Thank you. I see a few people commenting in the chat that used to work at Target and love hearing those behind the scenes here too.
Hiring for curiosity and model governance
Abigail, I see you had a question. Want to jump in next? Hi. So increasingly, as I get to be like an old person, people send me their resumes and they ask me for advice and I give them advice, but it's hard for me to tell how much of my advice is completely idiosyncratic. So like, yeah, like when you're looking for junior people, like what do you look for? What do you assess? What are your metrics? Whatever.
Yeah, I'd say the biggest thing, regardless of what's on the resume or experience is just the curiosity and willingness to go experiment and do new things. This space has changed so much and it's going to continue to change so rapidly. Their willingness to learn, to dive in with curiosity, build and connect and work with peers.
The one thing I really, that's kept me at Target as long as it has, is the diversity of talent around me. And so I think their ability to be curious and then reach out to people and work with others and pull in diverse perspectives. Those are the things I'm often asking about in interviews. That differentiates somebody from what's just on their resume. And so I think that type of, however you can show that in terms of your experience or the projects you work on or the type of products, models, things you're building, I think that just goes such a long way.
Because rarely are you just doing it yourself and rarely are you the one who's done the initial research, algorithm development or model development that was published originally. At least I don't see that at Target. I mean, I'm sure some of the bigger tech organizations, you know, transforming the world. But I think there's a lot of just how do you, you know, do more of the applied side and work with teams and dive in to solve real problems.
Thank you. Adam, I see you had a question earlier on model governance. Want to ask that now? Yeah, just going back to the earlier conversation about that one anecdote that shall not be named. I'm curious about, you kind of mentioned like, well, you know, we didn't really have good model governance practices and things. I'm curious sort of like what the evolution of that has looked like if you're able to share how models get validated or reviewed for correctness or bias. And then, you know, is that team sort of separate or is all kind of like in-house within data science?
No, it's all in-house. I would love to say we have built this full like MLOps ecosystem out and we have, you know, some unbiased ways of measuring, scoring, doing all of that. I would say we do a much better job than we've ever done. We do have a data governance team that helps us at least with the data pieces and making sure, you know, what people are using and consuming as features
and things are much more controlled than they've ever been. But as we think about like model scoring and the outputs, like we do that individually.
We often work within our teams to do the reviews. And I think sometimes that's good enough. But I think there's a lot of times where we could do better. And, you know, that's something that's top of mind for Target, top of mind for where we're going with the future of our science platforms, where a lot of those things are, you know, in our desired end state is built right into the patterns and processes that we're following as scientists. And so we're on a journey to do that.
I'd say our tooling, for those who maybe don't know Target, all of our internal tooling is almost all in-house, open source built. And so a lot of what it takes to stand that up is a combination of just the right engineering partners, the right scientists, and, you know, honestly, even the right leadership to build that governance and controls across. And we are committed to doing that, but it is a journey and it's going to continue to evolve.
Yeah. And you're not behind. I mean, I think every business is really giving deep thought right now about how they go about that. So thanks for sharing. Yeah, of course.
Working with decision makers
Funday, I see you had a question a bit earlier as well on working with decision makers. Want to jump in?
Yeah, thanks. Great talk so far. Yeah, I'm just thinking this is something I get to encounter every now and then. You have people who are supposed to be senior level and make, you know, all these huge decisions. And they have this way of sounding really smart when they have all these nice plots and charts and graphs and stuff in front of them that has been prepared and all laid out. And yeah, they sound smart. But when you listen carefully enough, you sense they don't quite get what's going on, you know, and it plays out in decision. Sometimes you have to show up again and explain, you know, what this actually meant was, you know.
So if you were to pick one area, you'd say you could help improve, like offer some kind of training for, you know, and this is not trying to make everyone a data scientist, you know, say, let's say, conceptual understanding of maybe probability theory or, you know, one thing, if there was one area, you'd say high level conceptual that they could benefit from to help them understand this thing, these things better. What would it be?
That's a really good question. I'll start probably even more simplistic. Like, I think the biggest thing I preach to most of my business partners and even team is, do you understand the inputs that are affecting your outputs? And do you really understand what went into those decisions or those variables that led to, you know, the other models or algorithms or all of that? And most of the time, what I find is a lot of hypotheses on what happened within those inputs. Well, somebody else owned that or somebody else knew that or that was already decided and I can't change it.
And in reality, there's so much of just, can you affect the inputs instead of just changing or overriding or, you know, biasing in between what you want to have happen? That I think it starts with, do you take the time to understand those effects of what happened upstream of you? Which regardless of science, I think is a really good behavior. If I start with just what has happened before I ended up having to make this decision or what I'm dealing with right now, I can usually start with a point of, I've used the word empathy a couple of times, but I usually start with a point of empathy to say, what did they deal with? What was constrained for them? What were variables that they weren't able to consider? What was the data they didn't have? And then I can work back even to their inputs and say, wow, if I actually change this other thing or help them understand that other thing, the effects all the way through this can be changed as well. And so that, I mean, fundamentally I think is just a foundational building block for anybody in science or operations or whatever that I think is really important.
Yeah, just real quickly. And the last thing is, you know, if you could pick between, let's say, understanding bias, for example, you know, or, you know, because very often people don't get it when you say, these are the limitations or these are the things you should consider. They just go like, hey, this is the answer. And then you go for it. You know, just one area, just one area, you'd say.
I would just ask, you know, why? Why is this a problem? Like, you know, I think asking that why and understanding it, there's oftentimes where people will come to us with problems like, we need data science to do this. And when you ask why and then you ask them to define it, sometimes it's not even a data science problem, right? Like you need a binary decision, you know, and sometimes like we can help you build that, like we can do that. But like that doesn't take any data science model. And so I think asking the why and then digging into like, what do you really want to accomplish? Sometimes you can really simplify the solution without even having to do the work.
I would just ask, you know, why? Why is this a problem? Like, you know, I think asking that why and understanding it, there's oftentimes where people will come to us with problems like, we need data science to do this. And when you ask why and then you ask them to define it, sometimes it's not even a data science problem, right?
Target's tools and open source
I see some love for the Target GitHub in the chat. And I know there's also a Target Data Science blog, right? Yeah, there is. There's a Target Tech blog too for a lot of our open source work. OK, cool. We will have to go find that too.
Lisa, I see you asked a question. I want to jump in here next. Yeah, I remember what I asked. I think, oh, there we go. Yeah, I'm just curious. I'm also a former Target employee that left back in 2006. But just curious, what tools you're using for your work currently? How has that changed over time? Did the set of tools differ from data science team to data science team, all that sort of stuff?
It does. I alluded to us being kind of almost all open source. And so, you know, we use all different languages. It depends what we're building, what stage it's in. I'm a big, you know, the tool has to fit the right problem what you're working on. And so we're not going to bias that as much as possible. I'd say tooling that I'm most excited about we're investing in right now is kind of is the MLOps like platform for our scientists. What is the right things that we build there? What are the right, you know, feature store model manager, all of those things. So I think that's something I'm really excited about. But we will have to unify that for all languages and everything that we do because we do a lot of different work.
You know, my team spends a lot of time in both JVM based languages. So Scala as well as Kotlin are two big ones that we use for like optimization and OR work. But then we do a lot of work especially in our forecasting space. And then we do a ton of work in Python, obviously. And, you know, using all kinds of different libraries and things like that that are accessible to the team. So and then we have our we have internal tooling that's built now for like notebooks and dev practices and all of that. But we we make sure everything ends up in Git. It's just the way that we kind of govern and do manage all code and versioning and all of that stuff as well.
So it's a little bit of spread, right? And it just depends on even what we're doing to deploy it. Is it containerized, dockerized? Like whatever it might be. We have a lot of different practices try to serve up the outputs as APIs. So various tooling.
I see a few questions on open source. So maybe I'll try and group them here. But Ben asked, what was the process like for establishing an open source program?
Honestly, that was before me. I don't know. I think, you know, there's been some review and scrutiny. Anybody else who was around maybe around those times might know more. But I don't know how we eventually started focusing more on open source. I know at one point we looked across a lot of our L&M costs and we were probably like, wow, we have several BI tools. We have several different modeling tools. We have, you know, different database structures. And, you know, what are these different things that we can bring together to minimize that? So I'm guessing some of that was part of that, but I don't have all of that.
Thank you. Tim, I see you had a question around the packages. Do you want to jump in? Yeah, just on the open source side, speaking of someone who's brought open source into a few organizations that were initially adverse to it. How are you controlling access to open source libraries that aren't developed in-house so that you can satisfy your risk management team and, you know, make sure you're not accidentally introducing issues with breaking changes?
Yeah, we do have platform teams that do host a lot of these libraries. Honestly, some of it's more of a pain for our scientists than I'd say it is for the governance side. We're pretty slow to sometimes make the changes and upgrade some of the libraries and packages and versions and things. But I'd say, you know, we do try to govern those. We try to have a, you know, V plus one so that we're managing, you know, not only the existing version of the library, but the next one and maintaining the one behind it as well. So that we do have the right change management in there. But the platform team does centralize a lot of that. And so the platforms that we work off of are hosting those and that those libraries do go through some governance as they're introduced to the platforms themselves. So it abstracts a little bit away from the individual scientists going to do all of that themselves.
I see a lot of other questions in the chat. Let me try and pull up here. OK, Frank asked a question a bit earlier and I might need you to expand on what supply chain DCs mean.
Supply chain optimization
I'm going to jump in, Frank. Hi, so I was referring to distribution centers. That was from a kind of an ops research view. I know it kind of borders the data science space, but I'm just curious if what kind of modeling techniques you're using, whether it's mixed integer programming or different things like that to solve some of these supply chain problems. Mainly kind of around making sure that your production or not production, but outbound operations from a supply chain center can adequately supply your target stores.
Yeah, we do. We have several different optimizations that we run all across from mixed integer programs to linear programs to just various optimization techniques. We try to synchronize those as much as possible and even simulating the variables and what those optimizations are up and downstream. So as we think about, well, just an example, we have a state of inventory. And as we think about that state moving throughout, the optimization is making those decisions and we can use those different optimization engines to simulate variables up and downstream. And so we feed stochastic inputs into those and then derive different outcomes from that, which give us scenarios. And then those scenarios bias what gets input again. And so we use that from purchasing to positioning to what gets placed in a store to what gets replenished. And those variables are all then built across and we maintain that holistically. And so it's a, you know, if you were to build it overall, right? Like you could think about it being one big mixed integer program all the way through the reality of that as we build the components in the building blocks and then try to share states as we maximize or minimize different functions.
Very cool. Thank you. That was pretty generic without getting into and exposing so much of our detail, but I got close.
Working with big data and R in production
Thank you, Jacob. I see you had a question about working with big data. Want to jump in? Yeah, I was basically just wondering for your R models that you put in production, if you're dealing with millions of rows or more, what libraries do you use to try to speed up the computations? Are you mainly using things like data table for preprocessing and avoiding the tidy burst altogether?
I don't necessarily know. Alfredo, you can jump on and try to share more. You know, we're doing a lot. Yes, more than millions of rows. Yes, different ways of speeding that up. We use Spark heavily for a lot of different things. We'll wrap things, right? So it just depends. Yeah, I did post something in the chat on that, but Spark is for, it depends on the number of rows, but for the billions of rows, Spark is probably best practices. If you're looking at 100 rows, you could use, you know, a future deployer. And then, like I posted, Mirai is a new package out of Japan that is very promising for speeding. What's that called again? Mirai. Yeah, when Hallie and team was here last week, we went to a fireside chat and he mentioned it to me because I brought this up to him. And it's pretty new. It's in the chat. It's Mirai. It's Japanese. I'm probably not speaking it well. Yeah. M-I-R-A-I. Thanks, Alfredo. I'll say that for anybody who's listening to the recording. Yeah, thank you, Alfredo, for joining from your vacation, it looks like. This is our current Minnesota winter, by the way. Yeah.
Open source in regulated industries
Okay, let's see. Greg, I thought you had a question about coming from a highly regulated industry. You want to jump in? Yeah, absolutely. I was just interested. I love to see when companies have like public GitHub repos and you're actually sharing, you know, some of the research that you had from coming from finance, highly regulated. I feel like our legal and compliance folks would freak out about anything even close to that. Even internal repos have been
a topic of conversation, right behind the firewall. So how do you kind of balance what you share publicly versus what you keep internally to keep your competitive advantage?
I mean, it's not just the regulated industries. Like we have, even before I could do this talk, Rachel had to delay me a couple of weeks because like our comms partners have to weigh in, our risk and compliance people have to weigh in. There's reviews on talking points. Like there's just, there is oversight to all of that.
I'd say we, especially with like tech blogs and the open source stuff that we contribute, like we try to, you know, we try to minimize, right? Like the IP exposure and all of that stuff. Our teams are always patenting work, which then, right? Legal is always involved with. So like, I don't know. I think across our team right now, we have like 40 pending patents. So like, even like, you just have to manage all of that stuff. And it just, it takes time.
But we're constantly trying to balance that too.
Gotcha. Okay. It makes me feel a little better. We're not the only one, at least. No, and it's not easy, right? But I always talk about like, what can we do to abstract certain parts of the way? So sometimes like even with code is like as much as we can, you know, just like software engineering principles, right? Like as much as we can decouple components, then it makes it easier for us to abstract away some of the IP, but still share and contribute open source where we, where it's possible. Just an example.
Book recommendations
Awesome. Thank you. Yeah. I know we were just talking about all the books behind you when you first joined, but I see Jared had asked, I see a bunch of books behind you. Is there one or several that you recommend we read?
My favorite leadership book is Team of Rivals. It's about Abraham Lincoln. It's pretty in depth. It's a deep read, but it is fantastic for anybody in leadership. It highlights how he built a team of diverse perspectives, challenging disagreements, even fights and arguments. And I just think it was, it's an amazing read.
And I think you just learn a lot from his approach and understanding and it's good for anybody. So.
Awesome. That's the first time I've heard that one. Thank you. The other one I like for business leadership is Radical Collaboration. If you're into that, it talks a lot about how to, to move diverse teams forward, even through, you know, the mucky messiness of different personalities and other things to you.
Social media data and Gen AI
Thank you. Runhua, I see you asked a question around social media data. Want to jump in?
Oh, yeah. And now I think the social media data is complicated, but it's very valuable. And also now we have a generated AI or, you know, operating AI able to plug in and be able to, you know, help my information or should your team use that or how to deal with ethical issues if we do that.
Yeah, my team doesn't necessarily work in kind of the social data channels. My peers and other teams, obviously, on like the marketing side, our digital side, use a lot of that, especially like Target has a Pinterest account. We have our own Twitter and all of that stuff. And so we try to use that data, especially as it informs different decisions and things we might do through the site or somebody's account, whatever recommendations.
I would say I don't know what that's going to look like with Gen AI. I'm not sure how all these large language models and what that governance and perspective will be as we start to think about the intersection there. I think we're going to have to. I think especially as, you know, fine tuning and other things become more and more inevitable in the broader Gen AI space. I think that stuff becomes really influential. And I think we'll have to figure out what that looks like. But I don't have that answer today. And I think we're just on the cusp of learning through that at Target.
Benchmarking against competitors
Yeah. Bill, I see you had a question. Do you want to jump in to ask it or do you want me to?
Yeah. And it was around, do you have a line on whether what you are doing generally and broadly speaking is similar to others in the space? So like Walmart or Costco or other retail organizations?
Sorry, Rachel, can you rephrase that? Sorry, the question was just around, do you have a line on whether what you are doing generally and broadly speaking is similar to others such as Walmart, for example?
Yeah, there's definitely ways where we're leading and definitely areas where we're following in different aspects of the work we do. We're constantly trying to benchmark competitors and understand different approaches to it. We do bring in consultants from time to time just to even give us an external perspective on where we can evolve and maybe look at things differently to remain competitive. So yeah, we do have a beat on some of that.
I think one thing you see in some of these areas of data science, especially data science and retails, we often share talent across different organizations. So there's people who my team has hired who are ex-Amazon or ex-Walmart or whatever else. And so we tend to leverage and build around those experiences as well. So yeah, we definitely have a good beat on where we're leading, where we have opportunities, where we're clearly trailing as well.
Career advice
Thank you. I always like to make sure I get to fit this question in, so I'll ask it now. But is there a piece of career advice that comes to mind for you, whether it's something you've given or you've received that advice that you'd like to share with us here?
I'd say the biggest piece of career advice, you know, I lean into empathy and humility. And so I'd say always assume you're not the smartest in the room. Ask a lot of questions and then be willing to adjust. I think as a leader, I have found more and more often I'm surrounded by diverse perspectives and understanding. And what I might start the team down or shift the team around is usually not where we're going to end up. And so I think it's great when you can step back, let other people lead you, and then you can evolve your perspective as a leader as well.
And so I'd say always assume you're not the smartest in the room. Ask a lot of questions and then be willing to adjust.
Communicating with stakeholders
Thank you. As we get to the end here, I know there maybe are some questions that I missed. So I just want to say, feel free to raise your hand on Zoom, just so I know that I may have missed your question.
But I do see there was a question in the chat a bit earlier that was roughly what proportion, I think Rick asked this, roughly what proportion of the data science team's work is forward looking, prediction versus backward looking analysis? And with respect to predictions, to what degree does your team use simulation or scenario analysis?
For simulation scenarios, well, in data sciences, I would just say most of our work is forward looking decisions. In data sciences at Target, we've separated out like our measurements, metrics, our traditional data analytics, statistics functions. Those are all moved into business functions. So like your traditional BI and all of that's happening outside of data science. But our data sciences team is all about the decision models and augmented decisioning.
And then just the question around how much do we use simulation? I alluded to us being on a journey of kind of creating a digital twin. We eventually would love to have simulation across everything from vendors and purchasing all the way through our stores and even through what we deliver to our guests. And so we're on our journey to have all of that build. So we use simulation at scale every day in almost all of our work that we do.
Thank you. I see your hands raised and I'm apologies for not being able to pronounce your name, but want to jump in.
I think it's Azway is your name here.
Nope. Am I frozen or are they frozen? It looks like you are definitely not frozen. Okay, I'm definitely not frozen. I will ask the question then. The question they were going to ask I see in the chat was, you discussed the challenge of effectively communicating to stakeholders why a particular data project is the solution to a business problem without overwhelming them. Can you share strategies for ensuring clear communication that resonates with stakeholders?
Yeah, I love to start. Maybe this is because I've spent time in supply chain, but I really think fundamentally starting with the steps of problem definition and breakdown allows you to then help build a bridge of like what data actually helps you solve that problem. It removes the assumptions on the data will tell the future or the data will help us change this or that. And I think when you just start to break down, we talked a little bit about the why, but when you start to break down, you're trying to get to this point. You want to understand this. And then you look at, maybe it's the defects that caused it, or maybe it's the changes in upstream decisions that impacted it, or maybe it's external factors, guest experiences, demand shifts, whatever it might be. I think when you start to break down the problems in ways that they can tangibly see and fit, then you can also meet them in that, which allows you to communicate how the data fits the problem as well.
How, and just leaning into the communication topic a bit more, how do you teach communication to somebody new on your team? Do you have any strategies for that?
Lots of practice, right? I talk about the immersion experience for my leaders, but that, you know, when I talk about leaders, we have principals who are, you know, mostly PhDs, deep technical experts. I also consider them leaders. They don't have any DRs, but they're leading the technical vision, architecture frameworks, you know, practices that we do. And then our leaders, our directors, and I think equally the way they communicate and engage is super important. And so I think the best thing is practice and exposure and coming alongside as we think about what we're doing and who we're sharing with. So know your audience and then practice with them.
Sharing ideas and deciding what to work on
So maybe one last question before we all have to drop, unfortunately, is just thinking about like all the ideas that your team comes up with or just new ways of looking at data. How do you go about sharing ideas or deciding what to work on next?
Yeah, there's a couple of things. So I talked about us being on a journey to experiment and moving faster. We have something at Target called our 50 days of learning. And so we make sure our teams have the space. If you think about 50 days in a year, it's 20% of your time. We try to make sure the team has that space to explore new ideas and also look at, you know, sharing ideas across the organization and sponsoring those things. So, you know, if you're in, you know, my team on the OR side, but you want to work on maybe more of a deep learning project or something like that, we'll use that 50 days of learning to have somebody go get experience in recommendation systems or in search or some other things like that. And I think that's one unique way we do it.
But then also I talked about the product model. I think the product model and having that business partnership and expertise allows you to also index on the value that your team is working against. And so the more value you unlock, the more investment you get, the more creative and space you get to go work on those unique, exciting problems and research as well. And so I think when you blend that empathy and connection to the value they want to unlock, as well as your curiosity and connection to the team, you end up being able to do more independently too.
Absolutely. Thank you so much, Brad, for taking the time to join us today and sharing your experience. Thank you all for so many great questions too. I'm sorry if there were any questions that we didn't get to, but there were so many good ones in the chat. Have a great rest of the day, everybody.