image: thumbnail.jpg
Data Science Hangout | Frank Corrigan, Target | Understanding the Impact of Data Science
The Data Science Hangout is a weekly, free-to-join open conversation for current and aspiring data science leaders.
An accomplished leader in the space will join us each week and answer whatever questions the audience may have.
We were recently joined by Frank Corrigan, Director of Decision Intelligence at Target.
6:38 - Problem formation - trying to find the unknown unknowns and bringing them to the business
8:20 - Integrating your data science team into your company's business objectives (ex. newsletter)
10:50 - What is the divide between a business analyst and a data scientist, in your eyes? How business analysts and data scientists differ
15:32 - What is the biggest mistake you’ve made in your role and what did you learn from this mistake?
19:15 - Onboarding new team members effectively
25:00 - The importance of motivating non-data scientists
26:42 - Resources for data scientists
32:30 - Challenges when using different tools across a data science team
49:28 - Analytical thinking vs critical thinking skills
57:30 - Embracing the 80/20 Rule & the importance of Focus Time
1:02:48 - Two frameworks to be more effective with stakeholders
1:06:47 - Rebranding to "decision intelligence"
1:08:35 - Quantitatively measuring impact from data science insights
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
► Join the Data Science Hangout Live every Thursday from 12-1 ET: https://www.addevent.com/event/Qv9211919
Follow Us Here:
Website: https://www.rstudio.com
LinkedIn: https://www.linkedin.com/company/rstudio-pbc
Twitter: https://twitter.com/rstudio
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to the Data Science Hangout, and welcome back to everyone who's been here before and to those joining for the first time. I'm Rachel Dempsey, and I think we all kind of know the drill by now, but we want to use this time to focus on questions that are most important to you all. So there's really no agenda for these meetings, and everyone's welcome to join in live or put any questions that you have in the chat. This week we're also going to add a Slido link into the chat, so if you did want to ask anything anonymously, that you have the opportunity to do so as well. And Rob, if you wouldn't mind putting that into the chat, that would be awesome. Just the one I sent you on Slack.
Is there a password that people have to use?
Nope, so there shouldn't be a password. They can just go there, and you can put your name there or you can choose anonymous. But also just want to share with everyone at the beginning here that we will share the recording of each Hangout on YouTube so that other data science leaders and aspiring leaders can benefit from the discussion. But without further ado, I'm joined by my co-host for today, Frances Corrigan, also known goes by Frank, Director of Decision Intelligence at Target. And so Frank joined a few weeks ago for the first one and is here today as our featured data science leader. And talking with Frank, he's very passionate about discovering new ways to think about the world and the communication of the work that his team does. So I'm really excited to hear some of your insights, Frank, and have the opportunity to ask you a few questions. Would you be able to kind of introduce yourself and share a little bit about the work that you do?
Yeah, absolutely. And I'll start off by saying I am happy to take the lead on the questions that come through the chat this week, but I found it over the past few weeks to be incredibly helpful and useful hearing other people's perspective about the work they're doing, how they're communicating their work and some of the challenges they're facing. So by all means, I'm as much willing to hear other perspectives as give you mine. I work at Target currently. I've been at Target for almost four years now. I build and lead what's called in data science teams. We support transportation within the supply chain. And I typically default to calling our data analysts technical problem solvers because effectively that's what they are, right? We're here to solve problems for the supply chain, make things go faster, spend less money doing it, make sure that the shelves at your local Target store are full of product.
And we just do it a little bit differently than the operators or maybe the other business analysts, right? We have a different toolkit and it is a bit more technical, right? We write code. We all know R and we know Python. We're not software engineers. I'll tell you that. We try to be diligent in our coding practices, but it is definitely not software engineering that we're doing. And then the other thing I'll say is we don't shun Excel or spreadsheets. I think there is a kind of this going on that Excel is kind of old and it's over there and people that use Excel are just business analysts. But I mean, if you're solving problems, having those kinds of tools at your disposal are also helpful at times, right? Right time, right problem. Yeah. I sometimes say that Excel or CSVs are the whiteboard of data analysis, right? How quickly can you prototype an idea and communicate it with your partners and your stakeholders in Excel versus trying to mock something?
I'd say the next level is getting a notebook out and doing it there. But ultimately, maybe that gives you an idea of how my team operates and where we're coming from, right? So problem solver. So we want to solve problems. We have this technical toolkit to do it with. With that said, what I love most about it and why I keep doing it after, right? So I'd say I've been doing this stuff for about a decade, past four years at Target. But I love the that moment when you are analyzing data or you're finding insight and your perspective just kind of changes from you thought the world was a certain way to wait, that doesn't make sense or holy cow, right? I didn't never really thought about it that way. And I think that that goes back to when. So in my past, I've taught a few economics classes at the college level. And it was my favorite thing in the world when a student would look, right, have that look on their face and say, Oh, yeah, I never thought about it like that.
Right. And being able to deliver that that change of perspective was awesome, because I truly believe that if you can take in multiple perspectives and really shape an opinion or a view about how the world is, and we're trying to get as close to truth as possible, you end up making better decisions, right? You make better decisions as an individual, your team makes better decisions, your stakeholders, it's good for your company, it's good for the individuals. And then those good decisions compound on each other. And you ultimately have a profitable company or I think a better life. So my favorite part about being able to do it at work is I find that it's practice, right? For me, my team, my stakeholders to use data to think about things differently, and in a better fashion. So ultimately, we can provide value for target in my current state. But then also, I want everyone that that I work with, to be able to use that skill on the decisions that they make outside of work, as well in their relationships with their finances, so on and so forth.
I truly believe that if you can take in multiple perspectives and really shape an opinion or a view about how the world is, and we're trying to get as close to truth as possible, you end up making better decisions, right?
Problem formation and staying connected to the business
Thank you so much, Frank. Something that I'd love to ask you about before some of the questions come in is that whole like problem formation process. And I know you said, like, there's a few different ways that you go about the delivery of that and methods that you use. I would love to hear more about that. I think something you had mentioned was like the weekly newsletter or happy hours. And I love that idea.
Yeah, so there's, as with everything, there's two hands. So on one hand, if you're a data or data science team, you can sit there, right, you can sit at your desk, look at the backlog, figure out when requests come in, you need this data pulled, or you need this analysis done or need this question answered. On the other hand, there is a state where you want to be thinking creatively and really trying as best as possible to find the unknown unknowns, and try to tackle those because if you can find some of those and bring those to the business, that becomes the true, in my opinion, the true unlock. And in order to do that, you need a little bit of credibility, or maybe a lot of credibility. And you need ways to for yourself to get more creative.
So two of the ways that that I try to do that in my daily life in my line of work are my stakeholders in operations have weekly business reviews or WBRs. I make sure that I attend those WBRs so I can listen to the supply chain operators or the business analysts talk about the problems they are seeing in their business, because that can give me a spark of, oh, man, I know I'm reading this book on regression, maybe I can connect in some of the methods I'm learning over there with some of the problems I'm hearing about on this call. And then if I'm able to ask questions within those WBRs, then people say, oh, man, Frank shows up or Frank's team shows up, and they always ask really good questions. And if people think, right, doesn't have to be an answer, if they think that you can ask good questions and make them think about something differently, right, they're going to invite you back. And then the second thing that I do is I write a biweekly newsletter. And it is, I do a few things, and I've experimented a few ways. But ultimately, what I'm trying to do is write about the industry and the state of the supply chain industry, connect that in with data analysis methods, and do it in a way that is educational. So anyone can read it, whether you're a supply chain operator, or whether you're a data analyst, and you can get value out of it, because you're kind of creating that bridge between the technical methods, and then how that applies to the problems you are solving. And I do those two things, because at the end of the day, when I bring a solution, or my team brings a solution to our stakeholders, right, I want them to listen, right, because if they don't listen, they're going to get implemented, right. And by doing this, I basically integrate me and my team into the broader objectives that we're trying to work on. And we create an environment and a platform where you have both people listening and talking, and then it goes back and forth both ways.
That's great. And how do people find out about the newsletter? How did you start that? Is it mostly other like people interested in R? Or how do you share?
It's a great question. When I when I originally started doing it at Target, it was totally grassroots. I put everybody that I had on the email distribution, it was all it wasn't BCC at first, right, I just put everybody out there. And at the beginning of the letter, I would say, Hey, this is why I'm doing this. If anybody else wants to be included on this newsletter that that you don't see their name out there, pass it along. And if they if they want to be on it, they need to ask me and I actually made it I had a sort of a strict policy because I really wanted people that were interested in it. So unless you explicitly asked me via email or called me, I'm said, Hey, can I get on your newsletter distribution, I wouldn't add people. So I never added people just because I thought they would be interested. I want it to be a total pull system. And it worked really well. And then I created a large distribution and now I have I it's like distributions within distributions of teams. So I think there's somewhere close to 200 people on my, my weekly distribution and it's really word of mouth that that it keeps growing.
Business analysts vs. data scientists
That's really cool. I see a few questions are coming through on the Slido as well. Okay. So there's one question there that says, What is the divide between a business analyst and a data scientist in your eyes?
Yeah, awesome question. So concentric circles, of course, but I'll explain it one way. And there's probably other ways to think about this as well. In my eyes, the business analyst is there every day, trying to respond to the questions at hand for the day or the week, or maybe the two weeks, most of the stuff that they are working on is going to be relatively one off, right? Hey, can you help me figure out this carrier problem this week? Can you help me figure out why a vendor isn't shipping on time this week? And then those insights or those discoveries, ultimately, we want to be able to do that in a repeat fashion. So I kind of see that's where the technical problem solvers can bring their more technical skills in and say, Oh, that's great that we found that let's do that over and over again. So right business analysts, more ad hoc one offs, and then the technical problem solvers, we're trying to scale it up and make it repeatable. With that said, the technical problem solvers also sometimes are working on ad hoc, but maybe we need to figure out how many yard spaces do we need in our yards, because we're going to have the highest volumes that Target has ever pushed through our supply chain network this fall. Should we get additional off site yard space that fits in yards? So yeah, again, concentric circles, but that's a little bit of the differentiation that I see.
Thank you. Kavan, and I know I pronounce your name differently every time. So if you wouldn't mind introducing yourself and asking that question, I see you put it on Slido.
Yeah, it's pronounced Kavan. Like, as if you were to take the L out of Calvin.
Of course. Yeah, I was wondering what your most unique problem you have had to solve on your team slash in your industry.
A unique challenge: last mile delivery from stores
Most unique problem. I think that one of the, so at one point at Target, I was working in our last mile delivery space. So last mile transportation, we worked a lot with UPS and FedEx. And one of the more unique things that Target has done in the past few years is starting to deliver a lot of our last mile e-commerce packages from our stores. And as part of that, you can imagine that UPS and FedEx in their last mile delivery networks weren't used to doing or picking up two, three, 400 packages from a bunch of different locations throughout a certain geographic network. They were very used to, there's a distribution center, there's a big truck here. We need to move it from here to your distribution center, and then you deliver all those packages from there. So this was kind of a new thing, not just for Target as a huge retailer, but also for our carrier partners that we work with. And I got to really work on the forefront of that.
I don't know if it is fortunate or unfortunate, right? Some days are, whenever you're working on something new, you have frustrating days and some days you have huge wins. So I'd say that was a pretty unique experience in terms of figuring out, especially in supply chain, sometimes we're dealing with a lot of different systems. We're dealing with a lot of different human behavior. And we, Target has 1800 plus stores around the country. And we were shipping out of most of those stores. So you can imagine all those stores operating sort of in their own ecosystem, but trying to solve problems of making sure those packages got onto those trucks, the visibility of saying, are those packages getting packed on time? All the way, are they getting picked up by the carrier on time? Are they making it to the carrier's facility on time? Trying to string all of that data together, which my team was responsible for, and then trying to understand, how do we fill in the missing pieces that we know aren't there? And we're not going to have a technology solution to see that for a year down the road. I would say that's the thing that comes to mind is probably the more unique challenge that I've had the pleasure of working on.
Seth, I see you have a question that you entered in as well. And I think maybe we copied over wrong because it looks very similar to the one that we asked before. So Tyler, just want to confirm. So let me double check on that. And I'll jump to the next question. Rob, you had asked, what's the biggest mistake you've made in your role? And what did you learn from this mistake?
I saw that one. So that's been brewing in the back of my brain. We all have a lot of challenges that we face. The thing that comes to mind for me is if you cannot find and hire the right people, your job is going to be that much harder as a leader, as a manager. So when I think about the most important thing that I have to do when I'm running a team is identifying the right talent and then trying to convince them to come onto the team.
In terms of mistakes, there are people that I've had on my team and they've done great and they've come onto the team, they've onboarded, they've gone through, contributed a ton of value and then left. There's people who have come on the team and probably realized this isn't the right place for them.
What I'm really getting to is when I think about the right talent, what does that mean? I heavily index on behaviors. And I always think that if you can find individuals that have the right behaviors, then the outcomes are going to follow from that. So when I think about behaviors, folks that are bold and intellectually curious, they have a hard time sleeping if they can't go find an answer or come up with a process to go find an answer. That's really huge. I find extreme ownership to be another winning behavior. I want people on the team who are going to show up and say, oh man, no, that problem happened. That's on me. I should have done something differently. Even when it's not on them, right? Everyone else in the room looks at them and says, that's definitely not on you. There's nothing you could have done differently. But just having that attitude of saying, right, I could change something I do next time, and that sense of control is really big. And then also I always talk about learning frameworks. I think that the people on my team should be learning constantly and not just, right, it's not just, hey, I have this project, I have to learn a new forecasting skill. I think it is really important for all people that work in the data industry to just be constantly learning and reading because that's, right, I referenced the unknown and unknowns before. I feel like that's how you can ultimately find those. You'll be pummeled with business problems all day long, but it's the people who are kind of going outside of the normal day and trying to get other information and bring that back to those problems who really thrive. So in terms of mistakes, no particular TM comes to mind, but there's been a lot of learning over the past 10 years that I've been hiring people and building teams to get to a point where I'm like, oh, yeah, it is behaviors. When I go into an interview with a potential team member, I'm going to index on trying to figure out what their behaviors are and how they will fit into the team.
I heavily index on behaviors. And I always think that if you can find individuals that have the right behaviors, then the outcomes are going to follow from that.
Do you think some of those behaviors are something that you can help teach them or it's something that they kind of have to come to the role with?
Let's say 50-50. Yeah, there's definitely things that you can learn, but some things seem to be intrinsic a little bit or maybe have learned over a long period of time. So like if it's something that you can learn over a three-year period, that's a long runway.
Onboarding new team members
Yeah. Sep, I'm seeing the right question from you now. Would you want to expand a bit on the question regarding onboarding?
Yeah, sure. That's a nice tangent considering you're talking about onboarding. I was just kind of wondering, I mean, as you're speaking about hiring and finding the right talent, part of that is once you do hire is onboarding, you have these newsletters. What's your process for getting people kind of up and running and obviously contributing in the manner that you hired them for, but also kind of being in sync with the rest of the team with everything else?
Yeah, for sure. So there are, I'd say, the two components that I get new team members to focus on most heavily. The first is developing relationships. So there's a lot of get-to-know-you's at Target. You come work at Target, you see those all over your calendar. Whether I've been here for a few years, because we have new people coming on, and especially if it's your first month in role. But because I do index heavily on the relationship aspect, because I think the relationship will help facilitate communication, I say, let's make sure that you are getting to know our stakeholders better and our partners. So I'll set up a lot of those and make sure those copies are happening. And then on the technical side, we have the technical tools and the processes that we use. It's not incredibly diverse. We do use R and Python. I know some shops who are relatively small will just be an R shop or a Python shop, but we have a relatively small set of tools that we use. So I do have team members, and I think I always have on any team that I've built, who are very inclined to teach other people what they know. So you know who those people are on your team and you say, hey, this is your partner for the first month, or a new team member. This is your partner for the first month. They're going to show you the tools that you need to know, where the data is that you need to know. And then they'll also show you what we don't have today, because we do have limitations in our capabilities. And then when I check in, something I do, especially if it's a relatively new, if we have someone coming out of school and it is their first job or their second job, they've only been in the workforce for a I'll make sure to have a, right, personally, even if, right, I have senior managers under me, but if they're coming out, I'll make sure to have a 15 minute check in with them every day, right? Especially in virtual world, they might feel a little bit lost. So I'll have a 15 minute check in every day and kind of say, hey, who are you meeting? What do you think? Did you, did you like them? Would you have another coffee with them? But then also, hey, what, what technical skills or what is what data set do you learn about today? Just to keep us aligned.
Interviewing for behavioral qualities
Great. Thank you. This kind of, this question kind of goes back to what we were just talking about before about some of the behavioral skills, but how do you actually interview candidates for those qualities? Like, do you use assignments as part of an interview process as well?
So personally, I have not used assignments. There's two, two different ways. So let me talk about the two different ways. So the first is I, I spent a year working at Wayfair and I think this is a habit I picked up from Wayfair because any, if you've ever gone on interview to Wayfair, you've probably been through a case question or a few case questions because they love that there. And I picked up the habit there and I feel like working with a potential candidate, no matter what level, we're just working through a case question, trying to solve a problem together and either do it on the whiteboard or you need to have a piece of paper and then you can show each other what you're writing down. You can identify a considerable amount of behavior and attitudes through doing that exercise with each other. And then also follow up questions is huge in the interview. So there's always the, let me ask you the first question and then if the candidate gives you an answer, always have a follow up question because I find if you can go a level deeper, that's kind of the more revealing segment.
So case questions I'd say are number one. Number two, we, I had a partner that would, he had a linear regression interview question and it would be, hey, do you know R, do you know Python? People that could do this in Excel, great. I don't think anyone ever tried to do it in Excel, but it was a relatively basic linear, build a linear regression question and then go through it and talk through the ahas, oh, there's heteroscedasticity in the model that's going to throw us off and have candidates run through that. And you can kind of see how they think. And it really, really wasn't about can you do the code, right? Can you call the problem? It was more about how they think and how they try to, to navigate around the problem at hand. And I think that kind of brought out some of those behaviors that I had mentioned before going through that process.
Just talking about interviewing like that makes me a little bit nervous.
Just talking about interviewing on this forum makes me think I probably have to rethink some of my methods now.
The newsletter and reaching non-data scientists
I see there's an anonymous question on the slide, oh, Frank, talking about the newsletter. So you mentioned you only add people once they express clear interest. Do you feel it's also important to motivate non-data scientists to see the light?
Absolutely. Yeah. And don't get me wrong. It's not like I don't advertise at all. I've said a few times I am very adamant about making sure I'm looped in with my stakeholders, making sure I'm looped in with supply chain operators. And I'm always alluding to the newsletter that I write. And so I always throw that into a conversation here or there. And the curious folks will say, wait, what are you talking about? What newsletter are you talking about? And I'll call it, I won't call it necessarily a data analytics newsletter. I'll call it my supply chain newsletter, right? Or my supply chain updates newsletter. There's a little bit of selling that I do here and there.
But yeah, getting both perspectives is super important. And to be honest, I haven't done the counts, but I would say that if you took my distribution list of upward of 200 people for the newsletter that I'm writing now, I would say it's about 50-50 in terms of supply chain operators and inventory analysts making those decisions, and then more technical data science, technical problem solver roles.
I feel like that's something that can apply to any role as well, because I'm thinking about the meetups and hangouts and things that I help organize. It'd be helpful probably to make a newsletter for our own team to say this is what's going on too.
Hugh, I see you put a question into the chat. Would you want to introduce yourself and ask that live?
Sure. Hey, I'm Hugh Welch. I'm a management analyst with the VA. I'm just curious if you have any particular podcasts or YouTube channels or blogs that you follow to keep up with your knowledge or discover new skills.
Podcasts and learning resources
I'm always looking for new resources.
Yeah. Well, let's see if I have anything new for you. Maybe it's all just redundant. But yeah, so every morning – we used to commute, so I used to be a podcaster on the commute, but now I have a two-year-old Labradoodle, and I walk him every morning. We go out for a walk, so I listen to podcasts. The ones that I have on my radar now, I'll give you three.
So number one is The Knowledge Project by Shane Parrish, which is more mental model decision-making. That's a big one. The Freakonomics network of podcasts, so both Freakonomics people I mostly admire, because again, that's all about asking great questions. Ultimately, you want to get data and answer those questions, but I find those podcasts get me into a great mindset to ask good questions.
And then the most data-y podcast I listen to right now, a few weeks I discovered Learning Bayesian Statistics. Yeah, LBS, Learning Bayesian Statistics, which is really nerdy and pretty technical, but I've always – right, I'm not there yet by no means, but I've always wanted to be a Bayesian, so I'm trying to get there by immersing myself in the terminology and the frameworks and going a little bit deeper. And that one's a ton of fun, the way that the creator does it.
I'm curious if there's any others that everyone else would want to recommend as well.
I see Mike put not-so-standard deviations in the chat there.
This is Daniel. I'm about to jump, but I just started listening to Data Skeptic, which I find good.
I would – if you just started listening to it, Daniel, I would recommend you go back to the beginning. I thought Kyle's first two seasons were fantastic. All of it's good, but the first two were – yeah, I couldn't get enough of it.
Yeah, the reason I started listening to it was a lot of – my problems are kind of like time series-esque, and I think his latest season covers time series, so that's kind of how I first got connected to it.
Cool. Well, if there are others, feel free to put them in the chat too. It'd be cool to put a list of them down. I see, Mike, you asked a question on Slido. Do you want to ask that one live?
Sure. I was busy adding podcasts to my podcast feed. Yeah, I'm interested in whether your colleagues are working off consistent versions of tools. So, do you say, okay, everyone is now
on R4.0 or R4.1, and everyone's got the same version, same set of packages, same versions of packages? I mean, how do you manage that when you're in a small group?
I'll be honest. So, I'll start off with a no answer, and then I'll back out from there. So, no. But the reason that we're – we're almost being forced to be better about it because we – Target is developing better tools. Two – maybe two weeks ago, I mentioned that we have a team called Dataminr, and Dataminr is now producing hosted versions of RStudio, hosted notebooks, both in Python and R. So, we are – that is helping us to align our versions and write. So, at any given time, people are creating new notebooks, new scripts. It's kind of forcing us, and that is looped in with our GitHub repository. So, that is kind of a lot – like, forcing us to align, which I think is the purpose of having great data tools, right? If you have great data tools, they kind of force you – like, they don't allow you to have bad habits. And if you have anything that you can teach me, like, I'm all ears on this one.
Well, I – I wrestle with this constantly. I work in pharma. We're regulated, and it worries the heck out of me what versions of things people have on their desktops, right? And we have kind of like a separate instance of R, which is the kind of fully qualified, you know, documented SDLC stamped controlled version. But still, I know that colleagues are going to develop code on their desktops, and then when it breaks, they're going to come to me as resident R geek and go, hey, Mike, why is this broken? And, you know, my kind of gut reaction is burn it all down, get the standard version, retry, and then if it's still broken, I'll help you.
But that kind of battle of reproducibility, you know, managing versions and, you know, even just trying to help colleagues is super hard if you can't guarantee what they're using. And actually, it's something that I'm wrestling with right at the minute.
I believe it. Yeah. So, again, when we, these hosted products that we have, that stuff is all built in, we can just share that stuff with each other. Yeah.
Anyone else? Oh, sorry, go ahead. The other thing I'll mention is that you mentioned you're in healthcare, you're in pharmaceuticals, and it's very regulated. I'm in supply chain, most of the stuff we do, most of the stuff we do at supply chain in retail, most of the stuff we do is not food related. So, we are nowhere near the level of regulation, dealing with regulation that you are.
Yeah, I mean, it's more intense at other ends of the business, right? But still, if we're submitting anything to the FDA, we want to be really sure, you know, that not just that we can stand behind what people have done, but we could reproduce it time and time and time again. And, you know, occasionally it does happen, they come back, you know, five years, 10 years later and go, you know, you did that model. Well, can you just explain this to us? Or could you look at this thing for us? And so, we have to be ready to kind of go, okay, roll back.
So, that's definitely a challenge. Yeah, thinking about can I reproduce these results from this model five years from now? If you can reproduce it six months from now, you're doing really well.
model five years from now? If you can reproduce it six months from now, you're doing really well.
Managing R packages across teams
On a similar note, I'm curious what a lot of teams are doing to manage different R packages, too. Or like, how do you determine which ones people can use across?
That's a good question. I used to say to colleagues that, okay, there's a kind of shopping list of packages that we roll out. So, we kind of said, here's an officially supported version of R and with that comes this set, right? Because that set is also going to be aligned to that version of R upon our regulated qualified instance. So, if you develop with that set, we know you'll get it to run on our production system. But we've, that was horrible to maintain, because it was off, it was about a thousand packages. You know, if you go for a number of packages plus all their dependencies. And so, if you're trying to push a thousand packages to someone on their first install, A, it takes a while. And B, they're going to be, if it breaks, then you've got a lot of mopping up to do.
So, now I've kind of, we're switching over to supporting RStudio Server Pro and we're saying, okay, this is where you do your development work. If you want to work on your desktop, here's the version we're supporting and here's a kind of get you started set. And, you know, we get that from RStudio Package Manager. But we're then kind of saying, look, your desktop is just a sandbox. You know, try stuff out. If it works, great. If it doesn't work, whatever. And if you get into a complete mess, then just, you know, blow it all away and start again. We are tiptoeing into using our env package, or at least I'm trying to kind of persuade colleagues to go down that route, so that at least over and above that standard set, if they get something for a little project over here, then I know that they've got a kind of reproducible little environment for that project. But the real problem is that I have no control over what people do. So, they'll just sit there and go, install dot packages. And I'm like, well, where did you get that from? I don't know. You know, what snapshot of CRAN are you running? I don't know. Where did it go? I don't know. And then they'll go, it doesn't work, Mike. And I'm like, of course it doesn't work. And then you've got to try and back out of that.
And that's where it gets tricky. Because I think some people, like there's a spectrum of folks from the folks who really know what they're doing and are careful and manage it well and can unpick things when they go wrong. And then there's the folks at the other end who really don't care. They just want to get work done. And it's kind of in the middle. You know, the folks down this end will take what you give them and will do what they need. And most of the time they'll get away with it. It's the folks in the middle who kind of just, you know, tinker. And then things break. And that's the hard one to support. You're not sure why that thing isn't working anymore.
Teaching coding best practices
I'm just curious for everyone here, too, if there's ways to kind of teach that, like coding best practices, like from the get-go or if people do that. Those kind of good hygiene practices.
There was actually just yesterday, there was a talk from Greg Wilson. Can you guys hear me? Yes, I can. And it was about design patterns. It was a really good talk and he's going to give it again next month. And if you join, you have to pay. But it's all going to donations for a charity that he chose. And he talked a lot about this. He talked about the challenge that expert developers in any language, they write at a higher level of abstraction. And novice developers, the gap is just too big because the novice literally cannot read that because they're not at that level of abstraction. So bridging this gap is a huge challenge. And he said, actually, there isn't really a solution there. But I think a lot of that comes from learning by example and mentoring and having somebody show you and explain that.
But I don't think there's really, what he mentioned, there's really no clear resource yet. I think he was kind of trying to get people to try and author that, that helps people bridge this gap, right? Like you've got advanced high level stuff and then you've got like a lot of really basic stuff, but there's nobody that's really showing you how to refactor and go from, or very few to really go from the basic to the advanced and why you would really want to do that. So that's actually a huge gap in the learning, I think.
Rick, thank you for sharing that. Would you be able to share the link to that in the chat? Actually, I have the slide open because I was still making notes on it. It was really good. But he's still developing it. So he's going to give, if you're linked with him on LinkedIn, he puts his updates there, or I guess also on Twitter. I'll put it in the chat here. So these are the slides from the presentation. And he talks a little bit, I can't remember exactly where it is, where he talks about the abstractions and going to higher level stuff, but it's inside there. And he'll give the talk again in about one month, if you want to join it. So it's a work in progress. And once he's finished, he'll make a video and then release a webinar or a video on YouTube or something where you'll see his comments.
Is BI part of data science?
Great. Thank you. So that's my shift here is a little bit, but I see an anonymous question that came in, but it's, my leaders always refer to our BI tools when asked about data science. I cringe as a shiny programmer. Do you consider BI data science? Why or why not?
I, so personally, I would say yes. I would say it's a step in the data science process. I haven't been able to understand why data science is taken as the building predictive models as, and then like, there's the BI before that, because how do you build good predictive models if you can't do feature engineering and where does good feature engineering come from? To me, that comes from the BI stuff, right? Exploring your data visually, trying to understand the real, in the real world, how did that data get created?
We could probably, right, someone could probably disagree with me and say, well, you know, like some of this deep learning stuff and the neural networks that are being created, you can just feed it a bunch of data and then it gives you good, pretty good predictions. But I don't know, at least in my world, a lot of it is you got to know your data. You got to know the process through which it's created. You got to know what it's telling you and what it can't tell you before you build a good model. So that's my opinion. If anyone else feels free to, or wants to jump in on that too, if I don't want to kind of shift gears too quickly on certain topics, if people want to add anything else.
Just for clarification, is the interpretation like BI tools like Power BI or just like other business intelligence tools? It was an anonymous question, but just as general business intelligence tools. I personally like it. I think it's just like how people, a lot of people use Excel and we were talking about this earlier, like it's an important tool and it's important for a lot of people as a stepping stone into more powerful tools. And it's like, yeah, again, the whiteboard of data science and it has its definite uses in the data science world. So yeah, it's not, it's definitely like still data science and it's definitely still a useful tool for people who can enjoy it.
So like not everybody speaks English. Some people speak Spanish and people who know how to speak Spanish will still use, still be speaking Spanish, even if they're not bilingual. So yeah, I think, I think it's important for people who are comfortable in BI to keep on using BI.
Curious of what the group thinks about situations maybe where the business decision makers are insisting on someone using, you know, the data science team using BI tools only. Does, do you guys come across that and how do you respond to that? Because that seems to be a thread, like why do you want to use these sets of tools when we've already invested a lot in insert, you know, the BI tool of choice?
From my perspective, the only time that I run into that is in the, the delivery or the output, right? They want output to be standardized from one platform or one, one view because, right, and standardization has its value, right? So being familiar with, you open your phone or you open your laptop, I'm going to take a look at the reports of the month or the week. It is helpful for leaders across, right, different teams and different parts of the company to have a common view. So to me, I can understand that component to a degree. I haven't run across, like, you need to do the analysis with this certain set, with this tool or these certain sets of tools.
No one else is bumping up against this.
Hey, so this is Hugh again. We're, this is kind of what we're actively living right now. You know, we've got certain officially blessed things that don't quite meet all of our needs. We're having to justify why we want to go to, you know, alternatives. And one of the things that's really helped us out with our leadership is showing value early, you know, getting something that can't be done in another way or is not easily done and the resources we're given. And then taking that and say, hey, here's how we can add value using these open source tools that don't
cost us anything. And that's usually kind of the big hook right there is this cost us nothing other than our time to build. But I will say it's still an uphill battle. And something that I've seen, I'm new to my company, but we're using things like Power BI to do a lot more complex data cleaning and data manipulation. And to me, that's not what Power BI is built for. You need to pull that data out, run it through R, you need to run it through Python, whatever it may be. It doesn't matter, just not in Power BI because you're going to slow your whole process down. You're going to, you're going to bog down, I mean, whatever server you're running that stuff on. So that's been our struggle from what I've seen is trying to get them out of the using the tool for the incorrect task, basically.
I've done something similar where I've, I've done both like the cleaning in BI, like Power BI and in R. And then I've just shown like my managers, like here's the difference. And they make their decision and they usually like focus on whatever's faster. So yeah, if there are two decisions and if you have the time, then like, yeah, just making both or making both, both the R version and the Power BI version. And then yeah, just seeing how they like either of those solutions.
Hugh, I'm curious, you mentioned building something that shows something that I can really only do this or do it well, using the tool of choice for you. Can you give an example of something like that would be?
Sure. Try and be brief. A lot of things we build here are specifically for our C-suite leadership, but they need to be able to get that 30,000 foot view and then on demand drill down into, okay, this is what I need to see. If we identify a problem, I need to get to that mid-level information and then hand it off in the same tool to frontline leadership to actually address the operational issues. I've experienced with Tableau, use that for a lot. I've used Power BI a bit personally to my frustration. And I haven't run into anything that lets you take in one cohesive package, go from very, very high level down through all the different stages to get to a very granular level with as much user friendliness as you can build into some of the things we do with the things we build in R.
Thank you. Unless people have anything else to add to this thread, I see there are a few other questions that came in too. And Rick, one that you just put in around differentiating between analytical thinking and critical thinking skills. Would you want to expand a bit on that question?
I guess I'm thinking about it because there was somebody just posted on LinkedIn a couple of days ago, somebody that's, I can't remember who it was, somebody that does things on data literacy and things like this. And then he posted, I guess to try and cage responses from his audience was, what do you do to develop critical thinking skills? But he didn't really define what he meant by critical thinking skills. And I realized I have my definition that I understand it and I put that into my own mental models that make sense. And I think it makes really good sense for me. But then I realized that actually these people had a lot of different ideas. So his suggestion to develop critical thinking skills was to be by yourself for a long time, I think. And I was like, I think you can develop meditation skills. And I think you can maybe develop creativity skills. I mean, I like to think a lot, but I mean, maybe that's critical thinking skills. I just think of questions and make connections when I'm just thinking by myself, and it's a lot of fun. But I think that doesn't necessarily, just thinking doesn't necessarily develop those skills. So the question is, how do you define the difference between analytical and critical thinking skills? Those terms get tossed around a lot, and I think they're poorly understood or defined. And is that important for you? And how do you actually evaluate that?
Critical thinking vs. analytical thinking
Man. Okay. So when I think about critical thinking skills, I think that is, in my mind, I layer in a component of curiosity and drive, because I think about my bicycle is broken, right? I'm going to figure out what's wrong with my bicycle. You try something, that didn't work. Then it takes you a second to think, man, what else could it be, right? It could even be a flat tire, right? What are the reasons my tire went flat? And what are the reasons why my tire went flat? And I look at the tube, and this is the reason why the tube went flat. It's broken, and certainly the amount of critical thinking that goes into that will hopefully lead you to not inflate your tire as much next time you go riding. And maybe you had to do some research on the web, or maybe you had a great friend tell you like, oh, if you over-inflate your tire, it's going to press down too much, like too much pressure when you roll over a rock, and it's going to pinch your tube and likely to get a hole. To me, like going through all those steps is critical thinking, right? So it applies to a ton of problems that we face in our daily lives and in work.
I will contrast that with analytical thinking, which is the ability or the tendency to apply these quantitative methods to try to help answer your question or solve your problem. So the one thing that jumped to my mind was in our space, we were trying to figure out, there's different types of distribution centers all around the country. Target has distribution centers. We have a bunch of trucks in our distribution centers. All the trailers and trucks in our, let's call them trailers, all the trailers in our yards are in different statuses, right? Some have product on them that needs to be unloaded to the DC. Some are being loaded so that they can go deliver to the stores. There are some that are, we can't use because they need maintenance on them, right? And there's this question, how much in each yard should there be of certain types of trailers, right? Because then once one of those types goes out of threshold, you can say, hey DC, this is getting too high. Like we got to bring that down. We got to work this. We got to get these out of the yard, things like that. Analytically, how could you try to solve that mathematically, right? And to me, someone who's thinking really analytically can say like, oh well, we can build a discrete event simulation model and try to figure out how the trucks are moving around. And once we get to a point where a certain number of trailers over X, that's your threshold number there because then we have problems. I feel like that is sort of, if you were to read a textbook, right? That's what you're going to get. To me, analytical thinking goes beyond that and that's where you start connecting other dots. So someone says, oh what if we try to use a decision tree to say like, there's a problem in the yard when the types of trailers gets to whatever number. And then you think about your top layer and then it splits, right? And then it splits again. What are those split numbers of your decision tree? Could those be your threshold numbers? And the answer is, right, we've actually tried this. We didn't have much of this, but.
I would counter argue that. Sorry to interrupt. I would counter argue that. I would say that that's creative thinking. You're thinking beyond the standard textbook toolkit, right? You're thinking, how can I use a tool that wasn't meant for this purpose but can handle that kind of data and it works and it's going to be a valid, all the assumptions are met and it's going to be a valid result. But actually nobody's going to really think about doing that. And I would say that's another component, which is the creative thinking on top of that, which I guess is part of the analytical. Would you agree with my critical thinking assessment?
Yeah, I think the way that I started to understand to put into a mental model for myself was that critical thinking is kind of like what you described it. Think about like all the ways that can go wrong and you're really thinking about the interaction and all the different components fit together. And when you pull this lever, how does it affect the rest of, and all the different inputs and all the different pieces working together. And so it's really systems thinking. It's a big picture approach. And the analytical, I would agree with what you said. Aside from the creativity part, the analytical is reductionist thinking, which is how many, many people are taught to think and are taught in universities and are taught when they learn programming or data science. You break something down to its component parts and then you just focus on understanding this one little tiny thing. I'm a biologist and you studied this one gene and that's it. Back in the good old days, you would have somebody study a single protein for 40 years. That was a long time ago. So they would dissect that protein for everything that they can possibly do. And then kind of revolve and say, well, we want to go back and understand how that protein works in the whole kind of system. So then I put that on critical thinking is the big picture systems thinking and analytical thinking is the reductionist component pieces.
And the issue that I realized with critical thinking is that we learn and we're taught and we live in a society that's very much focused on analytical thinking, right? I think like vitamins, right? Vitamins is analytical thinking. You can just take vitamin C. You can just take some protein. You can rub DNA on your skin and it's supposed to make you better. So you're just taking some one component that imagine this component alone is good. And then critical thinking says, well, you need to understand how everything fits together. And I think that's why critical thinking is really hard for technical people, because we learn by breaking things down, reducing it to their component pieces, but then putting them back together. I think that part is also, that's another part that's missing from the education, right? Like how do you see all these things really working together as a whole? And maybe that's also why we have a hard time talking with some business people that do see how all those things fit together because they see the big picture. And we want to talk about details and component pieces and they want to talk about the entire structure.
And I think that's why critical thinking is really hard for technical people, because we learn by breaking things down, reducing it to their component pieces, but then putting them back together.
I like it. I like the micro versus the macro distinction. And as you were describing, that got me thinking about Steve Levitt for economics, people I mostly admire. And he talks about his experience. And in the past, he read for about three months, he wanted to be a macro economist, but he just couldn't get his brain working that way. And I wonder if that is the critical thinking, but he's really, really good at the analytical thinking, right? The more macro stuff. Yeah, I found when I was studying, I naturally tended towards the big picture. And I want to see everything as a whole and see this big story. And my professors are really upset at me because they were like, what are you talking about this stuff? I was talking about combining evolution and all these different perspectives. And they were like, what is this? Just tell us about this protein. And I was like, that's so boring. You can just read that. To the paper, I'm telling you something really interesting here, like how these things fit together. And they weren't happening. So they really want to see the individual components. I think that really, that's why for me, kind of, I put it into that model. And a lot of my colleagues that were scientists were very, very, very focused on the fine detail things that they really couldn't get or see why you want to make this connection.
Scheduling time for big picture thinking
I like this idea of critical thinking. And I'm wondering, Frank, how, like, when you're prioritizing things that you have to do for different business units or projects that you're working on, how do you schedule time for actual, like critical thinking and looking at the big picture?
I mean, that's become, to me, that is something you have to practice, right? I would assume no matter what size company you work at, or if you work for yourself, you're, in our society, we have a lot of meetings back to back, right? We're always scheduling time for ourself to meet with people to work on a certain project. For me, over the past few years, I feel like I've been listening to these thought leaders that make it very apparent that if you're not scheduling time for yourself to be thinking about how the components of your team or the components of your stakeholders' problems are going together, you're always going to be reacting, right? You're never going to produce things that are truly game changing. So for me, and it's been a few years in practice, but I've been very adamant about scheduling time for myself and making sure that I disconnect and turn the laptop off. I also do a ton of running. I've been a runner my whole life, and that hour of my day is super, super important for that critical thinking, right? I go out, I don't listen to anything, I don't have headphones on. So that's an hour, right? I'm running through the woods, typically right through a trail, and I'm not distracted. And that kind of becomes, and whether that's at lunchtime, or that's before work, or whether that is at three o'clock in the afternoon, people that I work with now know that that is a critical part of my day. But it's hard, right? A lot of people don't have that luxury. I think I've created it over the past few years.
It's really helpful. I know that's something at RStudio that we kind of think about a lot, it's like creating more focused time for ourselves, and just talking with different people about what are their strategies for actually doing that when you're just constantly getting bombarded by emails or notifications all the time.
I know we're getting to the top of the hour here, so people have to jump, no worries. And Frank, if you have to run to another meeting, let me know. I'm rolling into lunch, so it's all good. Okay. Well, there's a few other questions, so I just want to make sure that none of them went unanswered. But I see there's an anonymous question here on Slido that is, what do you think is the mental model underused in the analytical field? That's a good question.
I'm going to preface this with, I'm going to say there's this philosopher, Derek Sivers. If anyone knows him, you know why I call him a philosopher. But a big thing for him is,
if he's going to a meetup, or an interview, or anything like that, he always tries to get the questions in advance, because he knows that his first answer is going to be the most important one. So, if he's going to a meetup, or an interview, he always tries to get the questions in advance, because he knows that his first answer is never his best answer. So, with that said, I'm going to give you guys the first thing that comes to my mind. Not necessarily my best answer, but the Pareto rule, the 80-20 rule. I think if you are coming out of college, day one, you're going to join really any team, for all intents and purposes. Let's say you're joining a data science team.
Continuously considering the 80-20 rule, where 80% of your results, or your outcomes, are driven by 20% of your effort, is really, really powerful. And I think if you have that 80-20 rule, and if you're a team, and you can embrace that, you can really start to understand that, let's tie this back to our last conversation, that focus time, I need that time off, so I can decompress. So, when I come back for the hour of work, my mind is there, and I'm in it, because it really boils down to quality over quantity. You can work eight hours straight in a row, and after the first two, two and a half hours, the quality of your work is not going to be great. And so, if you can figure out the way to really rest your brain and identify that 20% of time, that 20% of your time, if you can make that really, really good and focused, is going to drive a really quality 80% of your results and outcomes. And who was it, Tim Ferriss, that even wrote a book based on that? Does anyone else have an answer to that question? I would love to hear an alternative perspective. Doesn't have to be now. I was just going to say that, or alternative perspective, once we've all had a chance to think more about it, too.
Can always bring it up next week, too. I see one other question that's out there, unless I've missed any. So, feel free to jump in live as well. But there's another anonymous question that is, I'm curious about the framework you use to effectively communicate with your stakeholders. We have two that are really effective. One is that we have, we, as in, this is Greater Target, not just me and my team now, but Greater Target has built an in-house visualization tool. So, similar to Domo, but we built it in-house. We use that to run all of our weekly business reviews. And we build those in such a way that makes the experience is still a work in progress. My peer that's working on it every single month, huge new milestone. So, scaling up quickly. But the functionalities are repeatable across all of our weekly business reviews. So, I'd say that's big, right? Because we could have one leader be in one business review, walk into another one. It's the same experience. So, that's been huge.
And then the other is that when we do more technical analytical analysis, and in full transparency, my team is still catching up here, but I have a partner team that is always delivering their analysis in R Markdowns. And it is target branded, and it always looks the same, right? So, you have your document, you have your table of contents, you have your problem statement at the top, and then you go from there. And the visualizations are really crisp and clear, and sometimes they're plotly, so they're interactive. But at first, the stakeholders were like, why not Microsoft Word? Like, what's going on here? But over time, right, they keep consistently going back to that R Markdown and delivering it in the same way. And now the stakeholders expect their analysis to look like that. So, it's easy for them doing the analysis, right? They do their analysis in R, they use R Markdown, they deliver it, and their stakeholders are expecting that. So, I'd say those are the two patterns that I've seen that work really well in my experience.
And it really feels like it boils down to consistency.
Can I jump in for a second? Because I think about the, I don't know if it's really a mental model, that's why I didn't want to say, but I think it kind of, because it's something that you just mentioned, it's design, basically, it's good design principles, right? So, McDonald's doesn't change its color or its logo every month, because it's crazy. And so, we don't think about that, because we're not designers, but a graphics designer gets mental if the thing is one pixel off, right? Or if the thing is one shade of blue wrong, right? They're just like, it cannot happen, right? So, by having those templates and standardized, and also, if you think about writing, when I teach my students writing, I also think about, not to make it boring, but it has a clear layout, a clear standard, what information goes where, and even at the level of the sentence, right? So, the whole purpose is you can ignore the junk, and you focus on things that are different, and you don't have to waste the mental power to process that. And I think that these design principles, I don't know if it's really a mental model, but it's something that, it comes right back to the design patterns for designing software programming that I think data scientists oftentimes forget about. But those design principles are universally really good to help focus on the important things. Academic papers, right? Like, every academic paper I open up, I know, based upon my level of interest and what I need from it, what sections I need to explore, and what ones I don't. It depends if they also knew that they were supposed to write it in the proper way. So, then you can get it. That's why I blame the author if it doesn't work, because the structure is there, they just need to use it.
The "decision intelligence" rebrand
Frank, I know we talked about this very briefly on a prior session, too, but I think even, like, your title of Director of Decision Intelligence says something around the communication, and I was curious if that's something that you drove to, that title, or if that was from Target already, or how that came to be. Yeah, fantastic question. So, historically, and I say historically lightly, because I've only been with Target for four years, but it was similar in prior roles. There's this perception that data teams, whether you're a data science team, data analytics team, they do reporting. You do metrics and reporting, you deliver me my reports.
My decision to rebrand to decision intelligence was really to help my stakeholders change their perspective on what we do. So, I made, this was my decision, I think, like, maybe a lot of people, I fell in love with Cassie Kasarov and her posts on LinkedIn and her making friends with machine learning videos, and I thought, oh, this really gets the point across. So, I really made the branding change to decision intelligence, and it has really helped. And when I did that, I started off with all my stakeholders, and I said, hey, this is what we're doing. We're starting from decisions. Once we start from decisions, we'll understand default actions, and then we can really start to back into the numbers and metrics and models, but really, I'm coming, me and my data analysts are coming to the table, thinking about the decisions that you need to make, because we know there's problems that happen every day. We can't really do anything about it if we don't understand what the decisions are. And it was really a reframing of stuff that we always wanted to do, but really what it has allowed us to do is have stakeholders and partners that are now working with us that are thinking a little more broadly about the role of data teams, particularly my data team.
My decision to rebrand to decision intelligence was really to help my stakeholders change their perspective on what we do.
Closing the loop with stakeholders
I love that. I see Rob. Do you close the loop? Yeah, yeah. Do you close the loop with stakeholders who need, you know, your team's services, you know, following an analysis or what you hand over with them? Do you ever actually go to them and say, hey, did you like make a decision based off of this, or did you change the way that you're doing something? So, I think I'm fortunate in that all my stakeholders today, I'm, right, half of them I'm actually, I would say I'm friends with. So, we're very locked in, and I understand as they make the decision what's happening. So, I do understand when they actually do take an action, but I'm not really satisfied with just knowing that they took an action. What I'm trying to figure out right now, and if anyone has thoughts on this, I'd love to have a conversation later, is, if you make a prediction that there is a problem that is going to happen, or service level performance is going to deteriorate, that prediction is out there. You have stakeholders or operators or whoever that has access to that information. If you don't know specifically what they're doing, right, they could be having calls with people. They could be turning dials that you don't even know about in the background. Is there a way that over time you can log your historical data, right, your historical predictions, and then over time you can look at what actually materialized? Is there a way, and I really think it's regression modeling to say, based upon what we see in these groups where we are predicting performance deterioration compared to the whole, we can estimate that the predictions are having the intended effect. I think it's really comparing a few different groups there over time. As long as you are cataloging the predictions you are making, can you formulate that into a way to quantitatively measure the impact? So, to answer your question, number one, in my current situation, I'm still looped in with the stakeholders that, yes, I know when they are making decisions, but I don't know how to quantitatively measure it yet, which leads to number two. I'm trying to figure out ways to quantitatively measure it.
Yeah, waste comes to mind in all functions of work, but specifically data science. I'm wondering how much of a concern it is of yours that your team is producing stuff for outside stakeholders, and it's either not being used correctly or not being used at all, and at the end of the day, that's just the way it is. Is there a way that you can at the end of the day, that's just a waste of resources and money and time from your team?
You know what? I do share that concern, but it doesn't stop me. I know I continuously think about Netflix, and maybe a few years ago Reed Hastings said, hey, our failure rate has to be higher. If our success rate is at a certain level, that means we're not taking big enough risks. If we don't take big enough risks, we're never going to hit the next super hit TV show. I know that some of the stuff we're doing is not going to get used. I hope that it's not going to get used in the wrong way, and I hope we're being proactive and educative enough that it's going to be used in the wrong way. I know some stuff is going to be a waste. I don't have a great response for it. How do I make sure that it's not too much? Part of the answer might be the relationships you have. You talked about how close those relationships are with some of the folks, and so sometimes probably just through conversation they'll tell you or share with you that it was valuable, and there's value in it feeling valuable in addition to being able to actually measure the way you're attempting to. I think if people keep asking us to try, like if you keep getting invited back, there's got to be something there.
I think if people keep asking us to try, like if you keep getting invited back, there's got to be something there.
Closing remarks
I agree. Are there any other questions that we may have missed, or if anybody wants to jump in with a few final ones? Thank you so much, Frank, for joining and sharing all your insights with us. I was going to share the link to your blog, possibly in the chat, if that's a good way for people to get in touch with you, or what's the best way? Yeah, please do. I love writing. I don't know if I emphasized that enough on this call. I think it is really important for data people to write. When you're forced to write something, it really makes you rethink how your mind is working. It really makes you rethink how your mind is working. Yeah, I would love if people discovered my blog there, and then also LinkedIn is pretty easy to find me, as long as you're looking for Frank Corrigan rather than Francis. Awesome. Thank you so much, Frank. Really appreciate you joining today, and thank you so much, everyone else, for joining. We'll be here again same time next week. I'd love to see you there, too. Have a great rest of the day, everyone.