image: thumbnail.jpg
Communicating insights for impact | Noah Branham | Data Science Hangout
We were recently joined by Noah Branham, senior manager of analytics at DraftKings, to chat about marketing measurement, data science career paths, and communicating analysis to stakeholders.
In this Hangout, one topic we explore is Noah's three-trimester framework for career development, focusing on how he progressed from developing basic data analytics skills to mastering predictive modeling and optimization problems, and finally transitioning to a management role. He emphasizes the importance of understanding business problems before applying data techniques and how the skills needed for success as an analyst differ from those needed for a manager.
Resources mentioned in the video and zoom chat:
🔗 Speak at Posit Conf 2025 → https://posit.co/blog/speak-at-posit-conf-2025/
🔗 Cascadia R Conf → https://cascadiarconf.com/
🔗 Unified Branding with brand.yml → https://posit.co/blog/unified-branding-across-posit-tools-with-brand-yml/
🔗 The Motivated Speaker Book → https://www.articulationinc.com/the-motivated-speaker-book/
🔗 Model cards for transparent reporting → https://vetiver.posit.co/learn-more/model-card.html
🔗 Upcoming End-to-End Workflow Demo → https://pos.it/team-demo
If you didn’t join live, one great discussion you missed from the zoom chat was all about running, including marathon advice, favorite races, and being a proud member of the Slow AF Run Club 😂 It didn't have much to do with data, but it was definitely about being a community. You should really join us live to get in on the fun!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back to the Data Science Hangout, everybody. If this is your very first time joining, welcome. I'm Rachel, I lead Customer Marketing at Posit. And if Posit is new to you, we build enterprise solutions and open source tools for people who do data science with R and Python. And I'm still going to add this in here that we are the company formally called RStudio.
I've been a host of the Hangout for a few years now, and I am now joined by my co-host here, Libby.
Hello, everybody. I'm Libby. I'm a Community Manager with Posit, helping to foster our Hangout community here. And I'm also a Posit Academy Mentor. So I help people learn R and Python to do more with data in their everyday job.
Libby's in Texas. I forgot to add, I'm in Boston. So if anybody is ever wanting to grab coffee or in the area, let me know. Our office is in the Seaport area. But the Hangout is our open space to hear what's going on in the world of data across all different industries, chat about data science leadership, and connect with other people who are facing things similar as you. And we get together here every Thursday at the same time, same place. So if you're watching this as a recording on YouTube, and you want to join us live in the future, there will be details below to add it to your own calendar. And this is just a double check for everybody. Just make sure it's adding it from 12 to 1 Eastern Time. I know sometimes something's happening if people were like traveling when they added it, and it's staying in whatever time zone.
We all know how much fun time zones are to work with. But thank you so much to everybody who has helped make this the friendly and welcoming space that it is today. And we are all dedicated to keeping it that way. If you do ever have feedback about your experience that you want to share with me anonymously, good or bad, or any suggestions, Zoom will actually automatically launch a survey right when you end the Hangout as well. You can always reach out to me directly on LinkedIn too.
Yeah, same. And at the Hangout, we love hearing from you. It doesn't matter your years of experience, what your title is, what industry you're in, what language you work in or don't work in. And we really encourage you to connect in the chat. This is a space for all of us to hang out together. So briefly introduce yourself, add a LinkedIn profile link, maybe add your website, just some way to get a hold of you and show who you are. Because this chat moves quickly and then it goes away. So we want to make sure we stay connected after today.
Let's see, we are also going to try to grab questions from the chat. So this is the way that the Hangout works. It is an open discussion from the community moderated by Rachel and I. So it doesn't happen unless you ask questions. It's powered by you. There are three ways to do that. You can raise your hand in Zoom, we will call on you. You can jump in and ask your question. You can put your question in the Zoom chat. And if you can't use your mic right now, that's fine. You can just put a little asterisk next to it or in parentheses, like I can't use my mic right now, we'll ask it for you. And then there's also a Slido link that's going to be in the chat. Isabella just put it in there, where you can ask questions anonymously if you like. And we'll try to get to as many questions as we can. Sometimes we have way too many, but that's okay, keep them coming.
Perfect. I'm going to do one quick shout out here that the call for talks for PositComp 2025 just went out on Friday. So if anybody is interested or thinking about giving a talk, go check that out. Would love to hear from you all. But also, I love getting to use the Hangout as a way to meet people who are going to be at conference too. So as you get your tickets and let people know here in the chat, it would be great to be able to connect with each other to get to meet in person too. But with all that, thank you all so much for being here and joining us today. We're so excited to be joined by our other co-host for the day, Noah Branham, Senior Manager of Analytics at DraftKings. And Noah, I'd love to have you introduce yourself a bit here and share a little bit about your role today, but also something you like to do outside of work too.
Introducing Noah Branham
Yeah, totally. Thank you guys so much for having me. I'm a big RStudio fan, I should mention that. Yeah, so as Rachel said, I'm a Senior Manager at DraftKings. I've been there for about two and a half years. I oversee measurements of a number of different initiatives, but mostly our reach marketing initiatives. So TV, audio, and influencer, and then our monetization measurement efforts. So basically measuring how our external marketing affects our existing customer base and their types of play. So it's a lot of fun. We have a great time, definitely a very sports-focused culture. And previously I was in the insurance industry for a little over six years. I did some work on staffing optimization, which was a lot of fun. As insurance companies, staffing is like a major expense for them. And then for a period of time, I worked in pricing predictive modeling. So yeah, it's been, I've had a lot of fun, a lot of really interesting roles that's given me some pretty interesting experience.
And then in terms of things I do for fun, lately, I would say I've been really into running. I ran my first half marathon this last year, and then I really like to barbecue, actually. I just got a smoker in the last three years. So that's kind of a fun weekend activity for, it pairs well with football.
I love that. I just made the decision recently to run the Boston Marathon this year. So that's awesome. Connect over some running tips.
Noah, I think maybe for the audience here, it might be helpful to give a little more background on what DraftKings is as well to help us with some of the questions we're asking. Yeah, totally. I'll give a brief history on the industry. So sports betting really didn't become legal, I think, until 2018. So before that, FanDuel and DraftKings were the major players in the industry, but they had focused on a different type of game called daily fantasy sports. And so that was legal since, I think, 2012, where essentially you could set a lineup of players, and you enter a contest, and whoever's lineup has the highest points, but then you win that contest. And so that was deemed legal. But then in 2018, actual sports betting became legal, where you can bet on individual game outcomes and what we call player props, which are player-specific bets within a game. And I believe New Jersey was the first state, and a number of states, I think it's 26 total now, have been rolled out over the last now seven years.
How Noah got into data
Nice. Thank you. Yeah, we know DraftKings pretty well in our household here with our friend group, but I think it's helpful to just share with everybody too. Could you maybe share a little bit with us first about also how you got into data? And if you weren't working in data, what other career path would you have explored?
That's a great question. So in college, my kind of like focus had really been to try to go into politics. And so I had actually worked for a U.S. representative going into my senior year of college, and then worked for her briefly for a short period of time right after college. And it was a really good experience, but I realized pretty quickly for me that it wasn't for me. The ironic thing was during that time, I was in charge of dealing with a lot of different data and organizing data. And that's when I really started to get into learning Visual Basic for applications. And I was like, okay, I like this aspect of the job. I just didn't like the other components of the job. And that's what led me down to data analytics. And this was in 2009 and 2010. So analytics and data science weren't as prevalent as they were now. But coming out of the Great Recession, companies were just trying to be more data-driven and more efficient going forward. And I think that really helped spur on the data science movement. Obviously, a lot of data was being created just because of technology as well. But that opened up a lot of opportunities.
Marketing measurement at DraftKings
Thank you. I was also kind of wondering, you said that you measure the performance of marketing campaigns just like as an overall what your team does. Could you share maybe an example of a problem that your team is tackling now or that they've tackled in the past to help us kind of understand the day-to-day challenges that you see?
Yeah. So I can give an example. I will say I was told I can't go into too many specifics just for competitive reasons. But I will say essentially, companies like DraftKings or my previous employer, Liberty Mutual Insurance, spend a lot of money on marketing. And so not only is it a lot of money, but it's across different channels. And so a problem that a lot of firms, regardless of industry, are trying to figure out is what is the right amount of spend that they should be spending by channel? And so you can measure that through what's called multi-touch attribution, which is measuring direct effects of each individual marketing channel. So let's say you have an influencer on Twitter that is posting about your company. The attribution could be directly measuring the people clicking on the link and then coming to your site. Or it could be something like estimating the overall impact by understanding the synergies between the different marketing channels. And that is often done using regression and what's called a media mix model. And so that's much more high level. And so typically, firms will institute both types of methodologies to get a better understanding of how well their overall marketing is performing.
Thank you. I think that that helps a lot without going into nitty gritty details. There's also a sort of cheeky question on the Slido that I really liked, which is, are you allowed to bet on sports given your potential sort of insider info based on the research that you do? Or do you get to bet? That's a great question. Yeah. So we're not allowed to bet on DraftKings. So I can bet on a competitor app just because we don't set their lines. But we are not allowed to bet internally, no.
Thanks. Travis, I see you just put a question into the chat. Do you want to jump in here first? Sure. Thanks. So my question is about APIs. And DraftKings API is not open or particularly transparently documented, which is OK. I'm sure that's some kind of business decision that's made. I'm curious about what goes behind if that's anywhere near your domain, what decides whether or not to open such things up. So myself, I geek out on PlayerProp. That's fairly often. I've met some other people in this Hangout that do the same. And we all have this patchwork put together of finding out what's behind the DraftKings website and crafting our own APIs and things like that. But a paid version of an API would be amazing, even if people would pay for it. But why doesn't DraftKings, for example, pursue that, if you know? That's a great question. I don't know. But I can definitely ask. I can definitely ask.
Fun. Yeah. I mentioned in my question, you could get some influences, you know, like we have our Python influencers who are here who would love to be that person for you. OK, sounds good.
I love it, Travis. Pitching the data influencers. Way to go.
I see another Slido question that just came in. And it was, I'm interested in the mix of Python and R at DraftKings. What would you use R for? And what would you use Python for? Yeah, it's a great question. I would say in general, a lot of the teams use Python. My team specifically definitely has a good mix. I would say right now I have an analyst working on Bayesian models. And so we leverage PYMC for that. But a lot of our mixed effects models will use R for.
Small vs. large companies in data
Awesome. It looks like Abigail was just mentioning PYMC in the chat.
I had a question. I know that you, just from previous conversations, you've worked at bigger and smaller companies. And I have, too. Very small to very, very large. And I'm kind of wondering what the difference that you've seen is in working in data in small and large companies.
Yeah, that's a great question. I think that was actually a very big career decision for myself. I think when I started out, I really wanted to work at a smaller company and get a lot of experience in a lot of different areas. But I found out that I was, being a jack of all trades is being a master of none. And I think for me, that was good for a short period of time. But I ultimately started to get asked these very difficult questions by board members of these different startups. And I couldn't answer them because I just didn't have the skills at the time. And that's what really actually pushed me to go from a more traditional analytics role to go into data science at a larger company. Because not only were there resources, and by resources, I mean other really, really smart people who can help me out, who had PhDs and help train me, but resources in terms of being able to fund going to grad school and pay for courses and whatnot. And that really helped elevate my career. On paper, it might have looked like a slight step back, but it was really just a way of investing in myself.
it might have looked like a slight step back, but it was really just a way of investing in myself.
That was my next question. What approaches did you take over the years when you decided, hey, it's time for me to level up? Was it just grad school? Was there other stuff in there? Yeah. I think at first, I really tried to learn as much as I could on my own. I leveraged
data camp a lot. I talked to my friends who were a lot more technical than myself.
But I realized you can only go so far doing those things on your own. And so that was one of the benefits of going to a place like Liberty Mutual, where there was a lot of really smart people. We had a data science community, even within Liberty Mutual, that we had resources that we could talk through. And then one thing that I did that I actually really tried to focus on was I looked at a graduate school program's entire agenda, and I tried to learn as much as I could. So then when I went to graduate school, it wasn't as much of a chore. And some people in the graduate program struggled. But for me, it was more just filling in a lot of those gaps.
The three-trimester career framework
Thank you. Well, while we're starting to talk about your journey, I remember when we first met on our call, we were chatting a little bit about thinking of your career as trimesters. And I thought that might be a helpful conversation for the group here. Would you be able to share a little bit about what you mean by that? Yeah, totally. So I would say the first trimester was really just getting familiar with working in the corporate world. And so for me, a lot of that was how to conduct and lead meetings, how to get your basic data analytics skills in order. So understanding, at the time, the tool of choice was really more Excel for me, just based on where I worked. And then getting into SQL, learning how to create presentations, and actually share those presentations out and create clean decks. That was really the first trimester for me. And then that's where, like I mentioned before, I was starting to run into those struggles with how to answer really difficult business questions. So the second trimester was really getting into the data science world, understanding predictive modeling and optimization problems, and how to solve them.
And I think a big thing is understanding the business problem, and then knowing what type of technique you need to do to then tackle that problem. I think a lot of people really struggle with that. And I think that's a really powerful thing to be able to understand. So I spent a lot of time doing that. And I was almost pushed into management roles eventually. And so that's really my third trimester, which is the manager trimester, which I think was the biggest adjustment. I think what gets you success as an analyst doesn't necessarily get you success as a manager. So you have to reframe your goals and time management, and how you spend your hours during the day.
I think what gets you success as an analyst doesn't necessarily get you success as a manager. So you have to reframe your goals and time management, and how you spend your hours during the day.
Communicating analysis to non-technical stakeholders
I think that whole conversation on, or we could have a whole conversation on becoming a manager. But Rob, I see you asked a question in the chat just a few seconds ago. Do you want to jump in here next? That's Rob. Can you hear me? Yep, we can. Great. Yes, thank you. I was just curious if you had any recommendations on soft skills, the main soft skills you use, and any resources for those of us that are maybe more technical of reaching those who are on the less technical side. Because a lot of times I'll have what I think is an amazing presentation or an amazing outcome, but then trying to frame that for maybe somebody who isn't as far into the details.
Yeah, that's an excellent question. I think your problem is actually a very common problem that I call it when the rubber meets the road in data science. I've seen really excellent data scientists build fantastic models, but they haven't really been able to have the impact that they need to be successful in the role because they can't communicate their work. Somebody I think recently said to me, your analysis is 50% of it, but communicating your analysis is the other 50% as well.
Your analysis is 50% of it, but communicating your analysis is the other 50% as well.
So that's an excellent question. I think there isn't an exact resource that I have for you, but I can tell you what I do, which is I try to find someone on our team who is not familiar with our work, and I ask to spend like 30 minutes with them to just share our presentation with them just to get their feedback because it will all be new to them as it would be to an executive in my situation. So that's one tool I've leveraged going forward that's really helped. I think practice in general is just really important in terms of getting your information out. And then I think getting as much feedback as possible from your managers is really important as well.
Big wins and impact at Liberty Mutual
Excellent. Thank you. Hey, Marlene, I see that you had a great question. Are you able to hop in and ask? Oh, one second. Okay. Sorry. Definitely multitasking. Squirrel. Hi. No, thanks for joining us. And my question is more about the little and or big wins. Like what's the thing that you've done recently that you're just proud of? And it doesn't really matter if it's like launching a big model. Maybe it's just that you stood up to a work bully. I don't care what it is, right? I just kind of wanted to get a sense of somebody else outside my profession that has an accomplishment that they're proud of.
Yeah, totally. I'll actually give one from a previous role. When I was at Liberty, staffing was like a pretty large expense for us specifically on the operations side. So just to give you kind of like an idea of how it goes is every insurance company has adjusters who essentially handle claims. And the types of claims require different types of adjusters. And as you probably are aware of like the Los Angeles fires, for example, like those require certain types of adjusters to handle those claims. And there would be like a leaky roof from a storm in New England. And so we hire thousands of or that Liberty hires thousands of adjusters. So when there's too many claims in a given area, they have to hire essentially what are called temp adjusters. And there's like a cost both from like a customer service perspective and an actual like financial perspective. So those temp adjusters are not, they don't do as good a job and then they have much worse customer satisfaction ratings.
So essentially in year one, what we did was we kind of took over the model, I should say in quotes, that they use to staff in these different locations. And then we essentially built out an optimization to choose the optimal number of adjusters based on like a pretty fixed amount of what we call demand, which was claim volume. So that was like our big win for year one. And then our big win in year two was we were able to basically change the demand forecast by MSA. And so basically we were able to create a bunch of different scenarios for like different weather events and types of weather years, like El Nino and La Nina. And that allowed us to create a wide range of actual adjusters that we needed in these different MSAs. And that ultimately resulted in about $100 million of savings per year. So I think that's my biggest like impact win probably that I can think of recently.
Skills Care Squad and internal learning communities
That sounds cool. Thanks. Thank you, Noah. I wanted to ask about some of the things that I know that you've mentioned in conversations before and people in our call haven't been privy to, which are the Monetization Task Force and the Skills Care Squad. Is that something that you can talk about with us? Yeah. Leading those groups and what you learned? Yeah. So I can't actually get into details of the Monetization Task Force, but I can definitely speak about Skills Care Squad a little bit. So Skills Care Squad was actually started by another director, and it kind of is very much like an analytics community within DraftKings. So what it is, is we essentially have a group of people together that I lead, or I've led since last May. And we essentially create like a curriculum for analysts to learn. I should say managers as well. It's not just for analysts, but we focus on specific topics that we think the analytics org would benefit from. And we basically create a curriculum.
So we're actually just laying that out for the next year. I think in the last year, we focused a lot on technical skills, like k-means clustering, regression, data visualization. This next year, we're going to focus a lot more on presentation best practices, building out better presentations. We have a series called Communicating for Action, where we take different versions of a previous presentation, and we actually share each iteration. So there was the iteration of the presentation that was for our direct stakeholders, like manager, and my case, manager and marketer. But then that needs to get escalated to VP or executive level. We have a different version of that presentation, and we just talk through the changes that we made and how the results would change. Not necessarily the results, but the slides would change.
Thank you. As you can imagine, Libby and I love getting to talk about communities and hearing about internal learning communities as well. I was wondering a little bit more about how you got that off the ground and running. How did you actually get people to buy into it and find people from across the team? Yeah, so I didn't actually start it. A previous leader had started it. And we had been trying to rotate people off of the squad just to give people other opportunities. There was a lot of interest there. And so I was asked to take it over in May. And we actually doubled the squad size for 2025, just because there was a lot of interest. And we wanted to basically get more trainings out. But I think I was able to choose the team last year, just as I took it over. And it was a lot of people that I had trusted and I knew were going to be able to deliver on a lot of the technical presentations. And this year, I specifically wanted to rotate folks out, one, to get to know different people. But also, I wanted to balance it out with some folks who had skills in areas that we hadn't covered as much.
Informal knowledge sharing
Thank you. Arsanis, I see you just asked a question in the chat, if you want to hop in here next. Cool. Yeah, thanks, Rachel. And thank you so much, Noel, for joining us today. So my question was about the learning that you just described. That sounds like you're kind of putting together a formal learning curriculum. But I was actually wondering, what you all do, if anything, to encourage more informal knowledge sharing kind of across teams or within teams?
Yeah, it's a great question. I would say that varies at the probably director and manager level. So for my team specifically, I actually have specific names for them. But it's essentially an office hours. But I do it on Friday afternoons, and I always encourage people to bring a sweet treat. It's kind of like an informal thing. So I'll bring a cafe mocha and we'll review a model that somebody's worked on, and somebody will share a model. So I like to do informal brainstorm and knowledge share. I also kind of informally became this modeling person that people on the team can go to. So a lot of times, people will just reach out to me, and we'll set up time to at least brainstorm how I would approach the project. And then they'll come and have a check-in with me once the project's further along. I think your question's a really good one. I think instilling that openness among leaders. When I say leaders, it doesn't have to be a manager. It can be a lead data scientist or a lead analyst. And setting that up so that you have infrastructure so that people have the time sets your team up for success in the long run.
Tools and technology
Thank you, Noah. I wanted to swing back towards some sort of technical questions. I have one, and then I also saw that Sunday asked one, and it has an asterisk next to it so I can read it. And Sunday and I have similar questions. So my question is, does your team use Excel? We talk a lot about code-first analytics, but Excel is still super prevalent. So does your team use Excel? What do they use it for if they do? And then Sunday's question was on the other spectrum. Do you use languages like C++ or Rust for efficiency without going into specifics?
Can you describe any of that? No, we don't use C++ or Rust. Our team strictly uses Python and R and obviously SQL. I don't really consider that a programming language. That's a database language, but that's basically the infrastructure that we have. The way I approach things though is, and it kind of answers both questions in a way, is we can use whatever tools do the job. So it's not necessary to use Python for very quick analyses sometimes. So I'll use Excel for that at certain times. But then most of the time, because my team does a lot of modeling work, we will spend most of our time in R and Python.
Thanks, Noah. I see. Well, actually, maybe because we're already on the technical questions here, I know we chatted a little bit about how the team uses Snowflake. And I was wondering if we could talk a little bit about that, like where does the data live for the team and how do you connect to databases too? Yeah. I mean, we use Snowflake for pretty much everything. And we are bringing on Databricks as well for a lot of our work.
I just know that those two like come up quite a bit in the chat in these Hangouts. I don't have a specific question on them, but thought because you do use Snowflake, if people did have questions, that might be a good time to ask too. Jim, I see you had asked a question just a few minutes ago. Do you want to jump in here next? Yeah, sure thing. So my question, and this isn't a gotcha question, but a lot of product analytics companies are built around how do we get more users,
how do we keep them on our platform? And the first thing that comes to my mind and someone that does it really well is Duolingo, right? They have the streak and like I used Duolingo for a while and I was like, oh man, I'm just hooked on it. So my question to you is like, within your team, especially with it being gambling and an area that has to do with, there are people who have gambling addictions. How does your team kind of thread that needle between kind of creating this compelling product that keeps people engaged without necessarily promoting that problematic behavior? Or phrased another way, do you have patterns to identify where maybe people are engaging in problematic gambling?
Yeah, that's a great question. It's actually not within my team. We have a specific compliance team focused on that. They're called the Responsible Gaming Initiative. And so we actually have Responsible Gaming Analytics that build their own algorithms to get an understanding if there's a problem and the algorithms then flag that person. So that's a great question though. It is top of mind for all of us, right? And we want to make sure that everybody's using our product in a responsible way. Cool. Thanks, Noah.
Thank you, Noah. I saw Tony made a hilarious comment in the chat about Li Mu, the emu campaign. And it made me want to ask about your work in insurance. So do you feel like some of your work in insurance, because I'm previous insurance as well, kind of prepared you for the work that you do at DraftKings or do you feel like it's totally different?
Um, no, I actually think there's a lot of parallels. There's actually a lot of ex-insurance people at DraftKings. Oh, really?
Yeah, there actually is. I would say, it definitely did. I would say it all comes back to company culture and the culture of training that Liberty had. They had a really good infrastructure and community and resources to work internally to get better. So if you had that mindset of that growth mindset of always trying to learn and always trying to improve, it was an excellent place to go to work and basically get the most out of those resources. So I think the other aspects too, that I think insurance in general is not a sexy industry, right? But the one thing that it is, is it's a place that's been well-established and has absolutely massive data sets. So when I was building retention models on the pricing team, we were dealing with like 10 to 20 million rows of data, right? And so you just don't get that opportunity to work with data sets of that size at too many companies.
Snowflake vs. Databricks
Thank you. Okay. Going back to the Snowflake and Databricks stuff for a little bit, I see there was a Slido question that was, could you talk about when you would use Snowflake versus Databricks? It seems like there are similar platforms. So Snowflake, we primarily use as our SQL infrastructure. So if we need to pull data or write a query, that would be how we would use Snowflake for. Databricks, you can also use Snowflake, but you can also use R and Python. And because it's cloud computing, you can leverage parallel processing to deal with very complex problems and run the code in a much more efficient way in terms of runtime. There's a cost every time to use Databricks though. So we tend to be more conservative there just because if you're using a lot of cores, it's going to cost the company a lot of money.
Thank you. I wanted to just say one quick note in the middle of the Hangout, housekeeping thing, is I know sometimes people end up joining a little bit later because their invite is wrong on the time. And just want to remind everyone, so it's in the recording, that the Hangout runs from 12 to 1 Eastern time. And I think it's maybe adding onto a few people's calendars the wrong way. So Isabella or Curtis, if you could grab that ad event link, maybe people could just re-add it to their calendars as well. If you could just share that full link in the chat, that would be super helpful. It might have something to do with me using the short link, Curtis, instead of the full one. Perfect. Okay. Thank you so much. Sorry, Libby.
Oh, no, that's okay. I was seeing that Alan asked a great question. As always, Alan, are you available to hop in? I am. Hi. I find all the buttons here.
Hey, no, I'm really curious. And we've talked about this from like life science industry and pharma and stuff like that. But I think it's really relevant with your teams as well that they're doing lots of work and a whole lot of data science work kind of happens in public. There's conferences, there's talks, there's blog posts and all of that stuff. And that's a way where people get to feel like they're part of a community of practitioners and stuff. I'm curious in an environment like yours where you do have lots of secrecy, you've got lots of proprietary data, proprietary methods and stuff like that. Have you got any good ways to sort of foster that sense of belonging to a bigger community of data, people doing data work that helps your people feel connected and developing in a way that's not just purely internal to the team?
Yeah, it's a great question. I actually noticed that pretty quickly was that at Liberty, we would do a lot of external data science events. We don't do as much of that at DraftKings. And it's something that I can bring up that we should try and do more of. I do think part of it is we are setting lines and that is our competitive advantage. So some of that is the secret sauce. And it is a pretty competitive industry with FanDuel and some of the other major players. So I think there's some reasoning behind that. But I agree that's something that we should try and do more because I think it has benefits for recruiting as well. And it kind of generates interest in the product. But no, I think it's a great point. I know that doesn't directly answer your question, but I think it's an area of opportunity. Yeah. Yeah. Cool. Thanks.
The 80-20 split in analytics work
Awesome. Thank you so much. I see a great question from Slido and it's something that I have been curious about with lots of our guests as well. It says, of the analytical work you do, how much of it is the complicated analysis, such as modeling, in comparison to the importing, tidying, transforming, presenting, etc.? So basically, what proportion is the real complex modeling stuff? It's a great question. I would say it's probably 80-20. A lot of the work is tidying the data, doing exploratory data analysis. And I think that's one thing that I didn't realize when I was starting my career was that a lot of the work is getting the data in the right spot or transforming variables. And so I would say conservatively, yeah, like 70% or 80%. And then a lot of times what we're doing is we're taking a model and really just updating the infrastructure, so like the data sets, the variables, and then re-running that model with different dependent variables, depending on what we're trying to measure. Awesome. Thank you. I wonder, maybe some people can sound off in the chat, if they find that their work is the same, like that 80-20 split between mostly data cleaning and then much less the real complex stuff.
Communicating insights for impact
Well, thank you. I had another question, if I could. You mentioned earlier some people sort of hitting a barrier where they're not being as impactful as they could be. They're creating models, they're doing things, but they're not really moving a needle. So I was wondering if you could talk a little bit about how you ensure, maybe a couple of steps that you try to take or that you tell people to take, to ensure that their analyses are actually impactful and that they're getting used to make actual decisions.
Yeah, definitely. I think that's an area, to be honest, I've really focused on the last three years of my career, three or four years of my career, and I think it's really important. So I've seen, like I mentioned earlier, I've seen a lot of really strong technical analysts build out fantastic models, but the actual results didn't have impact because they weren't able to communicate it effectively. And that could be a variety of different reasons. That could be because their deck was sloppy and leadership saw it and looked at it for two seconds and was like, okay, I'm just not going to trust this. Or it could be because of a slight mistake in the presentation. Some leaders will immediately write off the presentation.
So I think the way to first deal with that is have very strict regulations around what goes into your presentation. So formatting, like little things like having all the fonts in the same size, having all the boxes on your slides be in the same format, and really being almost OCD about that. They were actually really strict about that at Liberty Mutual when I worked there. I accidentally, I remember, used the slight one shade off in yellow and everybody called it off on one of the bar charts. So I think really sticking to those makes it look just more professional and creates belief. And so I think I undervalued that earlier in my career, so that really helps. I think the second thing too is then having a sound story around your analysis. You can't just be sharing model results. It actually has to be translating that into business action. Like one of the best courses I took in grad school was actually called Strategy and Analytics. And it was basically just 10 case studies of the teacher giving us, the professor, excuse me, giving us regression results in a situation and then having us focus on turning that into a presentation and sharing it out and giving our recommendation. So it was less about the actual process of data science, but more just like actually turning it into impact.
You can't just be sharing model results. It actually has to be translating that into business action.
That sounds like such a great class.
It was great, yeah. It was great. It was one of my favorites. It was called Strategy and Analytics.
I love that. So when you're talking about using exactly the right shade of yellow in a chart, I just have to plug some of the work that our team has been doing on unified branding. So I just put a blog post into the chat, but making sure that you can use your company brand and your brand-yml docs and your Shiny application. And there's been a lot of work done there. So just sharing that in the chat. But Noah, a question that we always ask here at the Hangout is around career advice. And I'm wondering if there's a piece of meaningful career advice that either you've received or you've given others that you'd like to share with us.
I think there's a... Fortunately, I have really good mentors and leaders throughout my career, so I've received some very good pieces of advice. But I think the one that really stood out to me was if you are the expert in the room, and I mean on average, it's probably time for you to move on from your current role. Because there isn't a lot of people there to really help you grow and learn. I've been the opposite of the expert on certain teams, and those are situations where I might have gone home and I had imposter syndrome every single day, but I did learn the most during that time period. And so I think it was an important experience for me to have.
if you are the expert in the room, and I mean on average, it's probably time for you to move on from your current role. Because there isn't a lot of people there to really help you grow and learn.
Thank you. I see, Ryan, you had a follow-up question when we were talking about the strategy and analytics course. Do you want to jump in on that?
Sure, thanks. Yeah, just listening to that course, it sounds really interesting and useful. I just had mentioned that if you don't have a link to the course syllabus itself, which you probably don't, maybe you have some ideas of some resources that we could look into ourselves or something that could approximate that content. So I don't... I could try and find it. I don't think I have it anymore, but I can try and find it. I don't think I have it, but I can try and find it. But really all it was, it was data sets from a business, and he made a regression model, and he put it in a PDF and just gave it to us and said, hey, translate what this means for the business. You could do that. You could probably go on Kaggle, get data sets, build a model. It doesn't necessarily even have to be right for this situation. You can just assume it's correct. Personally, I would try to make the best model possible and then try and say, hey, what does this mean? And I think it's important to take a step back from the actual modeling process when you do that. I think that's actually a big... I don't want to say turning point might not be the right word, but I think a lot of analysts just get bogged down in that and forget about the business impact and the business implications of their work.
And so I think focusing on that is really important. We've actually had our team focus on building a proprietary code, both code, but then also data visualizations that help translate our model results to the business leaders that I think has really helped. And that's something that's taken time. I've had a lot of the same team for two and a half years, and so I think that really helps in help telling the story. That actually goes back to... That actually reminds me of something else. I think it really helps to actually go back to your exploratory data analysis to help tell the story. I think a lot of people are trying to lean on the model results, but if you go back to some of your EDA, that can actually help provide a visual representation. And a lot of people are visual learners, so that can really help tell the story.
Yeah, it's interesting. It sounds like you're kind of saying, don't worry so much about doing it right or wrong, just do it. And eventually it'll work its way into right.
Well, I think modeling-wise, in an actual professional setting, you want to make sure you have the best model possible. I was saying just for this specific example, if you were going to go get Kaggle data and build a model and then try to extrapolate what that means for the business, I'm saying just do that, just because the focus is not on the model for this exercise. The focus is more on translating what those results might mean.
Thank you for that question, Ryan. And I wanted to pitch just a little bit. I know that Acacia and Blythe from Articulation were on as guests last year, and their episode was fantastically fun, and they talked all about presenting and communication, and they just wrote a book. So I figured I would sort of put that in the chat for everybody if they wanted to pre-order that. It's called The Motivated Speaker.
But I did see that we had another great question. Kim, are you available to ask?
Yes, hi. Thank you. Sorry, first time here, so first question. Thank you. Yeah, my question was, you mentioned earlier on that you took some online courses yourself when you were looking to upskill. You did some data camp courses. And I guess I'm in a similar situation at the moment, and I'm wondering, I'm sort of weighing up formal qualification versus self-learning, and I'm wondering if you felt it necessary for yourself, not necessarily from an employer point of view, to have that piece, well not just piece of paper, but that qualification, or did you find the skills that you built yourself was enough in terms of, you know, skill set and being able to do the job in your domain?
It's a tough question because you have the trade-off between the two years that I was in school versus had I not gone to school, would I have been able to make up the difference in education? I think the degree gave me two things. It gave me a credential which opened doors, but it also gave me just personal confidence from getting validation and knowing that I talked to professors to get answers and to fill in some of those gaps. So I'd say for me, yes, it was worth it, but I think for some people, I think you can figure things out on your own. It really just depends on your skill, probably your degree, previous degrees, previous education in that situation.
Cool, thank you. Thank you. Sometimes during the Hangout when I see all these great questions, I'm kind of like creating this map in my head of how they all tie together, and so I see you were jumping, ping-ponging a little bit, but jumping back to communication, Kylie, I see you had a great question in the chat too. Want to jump in here?
Yeah, sure, thanks. Yeah, I was just curious because you've talked a lot about communicating with stakeholders, and one of the things that I have found most difficult when communicating with non-technical people is how you communicate uncertainty because you can have a really nice plot, but there's usually some kind of uncertainty in whatever you're projecting or extrapolating, but they only see a line, even if you have confidence bounds, and so it's a really unintuitive concept for a lot of stakeholders, yet it's really important to communicate, and I just wondered if you had any tips or tricks or you've had particular success with a way of visualizing uncertainty or talking about it?
Yeah, I think it's a great question. I think it's one in my situation where it was actually not, yeah, somebody actually just said it, all models are wrong, but some are useful. That's a famous quote from George Box, and I think I actually try to reference that in a lot of my presentations that models are essentially just guessing using statistics, and a lot of the situations that we're in, there'd be no alternative way of measuring it. So, you have to first have buy-in from your leaders who are not in data science that modeling is the solution, and I think one mistake I guess I would say I made earlier on was when we presented model results, we presented basically the average results, like on average, the model says X, Y, Z. We've gotten more now into range of outcomes, and I think the range of outcomes has helped build more confidence in saying like, hey, the model outputs a range between this value and this value, and on average, we think it's this, and I think that's helped get buy-in from a lot of leaders rather than point estimates.
We've gotten more now into range of outcomes, and I think the range of outcomes has helped build more confidence in saying like, hey, the model outputs a range between this value and this value, and on average, we think it's this, and I think that's helped get buy-in from a lot of leaders rather than point estimates.
Great, thanks. I also think somebody else said something in the comments that I thought was very good too, is starting with the so what, and then it's almost kind of like reverse engineering it. You start with, hey, this is what our model says, and then you can get into some of the details from there. I do think you can lose higher level leaders if you start to get into the minutiae of your model, but I keep all that information in the appendix, and I oftentimes reference it as the presentation goes on. I had one instance where I was presenting to our CMO, and we got into a relatively complicated concept, but I had one box plot that was like there to save the day to help explain the difference between two groups that we were looking at, and so that really helped in that situation.
Model cards and large data sets
I wanted to add a quick shout out to another event that's coming up and thought this might be helpful for people. I just learned what a model card is. I don't know if a lot of people know that term, but it's a way – it's a framework for organizing and presenting essential facts about deployed machine learning models, and Julia Szilagyi is going to lead a workshop demo on it this month, so that's January 29th that she's leading that, so I just put the link into the chat there as well. Let's see. Libby, do you see any chat questions we have missed?
I see a few that it looks like we could circle back to, and one is from Saul. Saul, are you still available to ask? Yeah. Thanks, Noah, for all the information. I think one thing that I'm interested in is working with such large data sets, there's always these segmentation possibilities of you can slice things a thousand different ways. Do you have any recommendations based on your experience on how you go about finding meaningful trends and patterns? Because what I get into is I get a lot of analysis paralysis, and sometimes I don't even know where to start when there's millions and billions of rows.
Yeah. I don't have a good answer, to be honest, and the only reason is when I was dealing with a lot of these large data sets, we had the technology to be able to use all of the data, and now at DraftKings, I focus more on aggregate data sets, so I haven't really had to run into that issue, but I know there's a variety of different techniques on dealing with large data sets that I definitely used in graduate school that you can use, like resampling techniques to take a look at. That's a good point. Thank you.
Becoming a manager
Thank you. Noah, going back to our conversation about the different trimesters of your career so far and on going back or becoming a manager, I thought it would be helpful to talk a little bit more about that. What was maybe most surprising to you about becoming a manager, or what did you learn through that process? I think the most surprising thing was just the volume of meetings and how your day-to-day changes. I went from basically in a world where I would have maybe one or two meetings a day, and I could just sit and build models, to a world where I had meetings from 9 a.m. to 5 p.m., and I had a bunch of people messaging me all day needing stuff or asking questions. I think that change was just very different.
My ability to spend time on modeling went down significantly. When I work on models now, it's typically before work or after work or just in my personal time because I'm interested. Then I think from there, that was a big surprise. I think that forced me into time management, forced me into really thinking about what it is that I want to be doing on a day-to-day basis, which has changed my responsibilities at work a little bit more to just focus on modeling now, as opposed to some of the more traditional analytics that requires more day-to-day work with our stakeholders.
Thank you, Noah. I wanted to ask, while we're on the topic of management and being a manager, are there any things that you keep in mind when you're trying to create an environment for your team to grow or an environment to foster curiosity or foster innovation? Even if you haven't put them into practice yet, what are some things that you are thinking about? I know that you put a lot of thought into being a good manager. Yeah, absolutely. I think I was pretty nervous, to be honest, to be a manager for the first time. I read a bunch of management books. I think a lot of the data management books are really interesting, but a lot of those management best practices were created before analytics and data science were a thing, so you have to apply your own spin to it. I think first and foremost, you have to have management principles. You should really, really think about those principles and how you can apply those principles.
My first job as a manager, I had very junior employees, and so my principles were very different in that role. My number one principle at that time was set my team up for success. If I ever had a situation where I was at a crossroads, I would think back to that principle. How do I set my team up for success? What action do I take to set up my team for success? Some of the time, that would be like, okay, maybe I shouldn't get involved as much and let this analyst who's trying to become a senior analyst take that work on his or her own to prove that they can do it on their own. Earlier on in that tenure, I might have said, hey, I have to step in because this is a junior employee. I think deciding on those principles is really important, and you're going to have different principles for different situations. The role that I have now, I have all senior employees under me, so the principles are not about setting up for success. These principles now are much different.
Thank you. Oh, sorry, Libby. You go ahead. No, I was just kind of wondering what, as you're switching into having more senior people underneath you, what is the main thing you think about for them? Is it career growth? Is it instead of setting up for success, what is it? It's different in this situation because of business need, but it's also different because while they all might be at the same level, their technical expertise might be different, and what they want to do in the long term might be different. I have some employees who've been only at DraftKings their entire life, or their entire career, I should say. For them, they're going to benefit most from my non-DraftKings experience and learnings from that. Then I have some who the work for them is just very easy, and so I just need to keep them interested by giving them more and more challenging projects. It really just varies person-by-person in this situation.
Keeping a career journal
I love that. Thank you. I know we have just two minutes here, and maybe one more piece of career advice type question. I know when we were chatting, you told me something that you do where you keep this detailed list of things that have happened over the last 10 years, and I keep thinking back that I wish I was doing that. I was wondering if you could just tell us a little bit more about that and how that's been helpful for you.
A lot of it, honestly, is what not to do. I'm a little bit of a cynical person, admittedly, so what I've noticed is when what I've noticed is when somebody makes a mistake, it sticks out to me, so I try and keep a note of like, hey, this didn't go well. Other people in the company were upset about this. There was a lot of internal talking about this specific thing, so I'll take note about, hey, this is what not to do. A lot of that is also in terms of management. If I had issues with the previous manager or I didn't like how something was communicated or something like that, I'll take note of how that communication happened just so that I don't do that in the future. Then I think back to what I have enjoyed in my roles over time, and I think about this periodically. A lot of times, this will be when I go get back from a vacation and I'm fresh and I'm just thinking, hey, what makes me happy at work? I try to just take a note of that. I don't do that every single day, I would say, but if something really stands out, I'll add it to my list. That's one of my favorite things to do coming back from vacation, before you open your computer. What is in my mind right now and what do I see as important and what makes me happy? Now I'm thinking all of those should be in one place together so I can go see them all.
That's one of my favorite things to do coming back from vacation, before you open your computer. What is in my mind right now and what do I see as important and what makes me happy?
Thank you so much, Noah, for taking the time to join us today. Thank you all for the great questions as well. It's so nice to see you all. Yeah, thank you guys so much. Really appreciate it. Take care.