image: thumbnail.jpg
Bayesian MMM and silly side projects | Ryan Timpe | Data Science Hangout
We were recently joined by Ryan Timpe, a Lead Data Scientist, to chat about Marketing Mix Modeling (MMM), silly data science side projects like brickr (see the link below to his 2020 posit::conf() talk), the benefits of working out loud, and using LLMs (Large Language Models) for data science coding (something he gave a whole talk about at posit::conf(2025)).
In this Hangout, we explore a bit of what Ryan does, including Marketing Mix Modeling (MMM) and helping business partners better use their data. He shared that his work utilizes Bayesian methods to manage the complexity and high seasonality of his data, especially the Q4 holiday-buying spikes seen in most retail sales data. Using Bayesian priors helps keep the models within constraints and prevents overfitting. The foundation of the internal MMM package that Ryan developed uses tidy models, and you can check out the link to his 2023 posit::conf() talk to hear more about his model pipeline! The theme here is that Ryan gives lots of great talks - go check them out!
Resources mentioned in the video and zoom chat:
🔗 Ryan Timpe's brickr GitHub Repository → https://github.com/ryantimpe/brickr
🔗 Ryan Timpe's 2020 talk on learning R with silly projects → https://youtu.be/oOG-aXP_ICI
🔗 Ryan Timpe's 2023 talk on model pipelines → https://youtu.be/R7XNqcCZnLg
🔗 Ryan Timpe's 2025 talk on learning from LLM pitfalls → https://youtu.be/vJrIahZWCw4
If you didn’t join live, one great discussion you missed from the zoom chat was about the importance of working out loud and publishing (even imperfect) data science projects. Attendees emphasized that overcoming their inner critic and sharing work helps with communication and professional development (but can be super challenging still). One participant shared that they got their current role by creating a fake MMM project with fake data, but that it showed their interviewer they could take initiative!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Timestamps
00:00 Introduction of Ryan
04:41 "What kind of MMM do you guys do?"
07:01 "What was your favorite silly project that you did?"
08:06 "What is brickr?"
11:11 "Do you think brickr helped you get the job you have now?"
14:23 "How do you scope your side projects?"
17:42 "What do you recommend for setting up a Python environment?"
21:16 "When do you recommend not using LLMs, and why?"
24:19 "How do you shut up your own inner critic?"
28:00 "What kind of data do you use for your LLMs and models?"
30:32 "Do you use open source MMM package like Meridian or Robyn?"
35:13 "How do you communicate complex modeling nuances to non-technical stakeholders?"
38:06 "Does your love for LEGO help you build code better?"
41:00 "What drives priority for what your team works on next?"
43:21 "What resources do you recommend for learning marketing analytics?"
44:46 "What tools do you use for Bayesian analysis?"
45:50 "Was your career move to the LEGO Group intentional or accidental?"
47:49 "What is your end product delivery?"
49:21 "What tool do you use to implement a Bayesian MCMC model?"
49:52 "How do you make model-based reporting faster and more automated?"
51:31 "Do you have a piece of career advice that's meaningful to you?"
Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
Can't wait to see you there. I would love to go ahead and introduce our featured leader for today, Ryan Timpe. He's a data scientist at the Lego Group. And you might know him from his hilarious conference talk in 2020, where he talked about learning R with funny side projects. That was my first introduction to Ryan. And then I've gotten to see him talk live in 2023. And he also gave a talk this year in 2025. And I will have all those links in a minute. Ryan, thank you so much for being with us. And it would be great if you could introduce yourself. Tell us a little bit about what you do and what you like to do for fun.
Awesome. Yeah, thanks for having me. Yeah, so I lead data sciences at the Lego Group. I've been there for six and a half years. So right before COVID and it feels like it went by super quickly. Everything I do is around marketing effectiveness and measuring how well all of our marketing and TV shows and everything that we do to try to get you to buy Lego sets, how that works and how successful that is. And it's a ton of models in the backend. And then it's a ton of working with the business to interpret those models and then action on them. Before that, I had like 10 years in the tech consulting firm doing a lot of macroeconomic forecasting and market sizing and survey analysis. And this was kind of before data science was a real thing. So eventually I just changed my job title to data scientist and became real somehow.
Besides that, I live in Connecticut, work in Boston, have a fluffy dog who's looking at me while we're not walking right now. And yeah, free time. Again, it was those silly data science side projects that I did a lot, especially a few years ago to help me learn a lot of things. More recently, just a lot of still doing that a little bit, a lot of Lego building, of course, because childhood me was obsessed with it and still I am. And then when I hit 30 a few years ago, I joined an adult gymnastics class and that takes up all my nights and it beats me up. But it's really fun and trying to do things I should have started doing as a really young kid, but now trying to do as an adult. So my body's always in pain, but I have so much fun doing it. I was going to say, have you taken out like an extra insurance policy, accident policy? That's probably a good idea. As a former insurance agent, I'm like insurance, insurance, insurance. No, but I'm very slow to progress because adults have fear receptors and I know that my entire livelihood is my brain. So falling on my head terrifies me. So yeah, I'm very slow, but I love it. So basically five nights a week, I'm at gymnastics. I love it. Yeah, we're not made of rubber anymore once we get past a certain age, right?
Audience questions and MMM explained
Well, I would love to remind everybody that you can ask questions in Slido and we already have so many questions, Ryan. I think that you've set some kind of record for the most Slido questions asked before we've even hit the 10 minute mark because there's like more than five in there right now. So why don't we go ahead and get started? I'm going to go hop over to Slido. I have an anonymous question, but it does have a name on it and it is Saptarishi. You had asked a question. If you are available to ask it live, Rachel can help you unmute.
Oh yeah, fine. Okay, I'm unmuted. Hey, I'm so happy I joined today. I did not know you also work at Lego, so this is fantastic. I'm a huge Lego person as well. Wait, let me see if I can turn my camera on for a second. Let me just say right now, officially, I'm talking about my job as much as I can and officially I'm not representing the Lego group today. We're very, very conscious about like brand identity. So I'm here as a data scientist, but yeah, definitely can talk about that. I do work in MMM as well, so this is really fantastic. What I was wondering is what kind of MMM do you guys do? Is it more like the Asian frequentist and how do you guys go about it? Like what's the usual workflow? And we should say like what MMM is for people in the chat. Oh yeah, sure. Let's start off with that and then more in the details of like what your MMM pipeline flow is like.
Yeah, so MMM is Marketing Mixed Modeling. It's an econometric equation to try to relate every type of marketing inputs to how much of sales or brand health or any of your KPIs you can allocate or attribute to each marketing lever. We definitely are more in a Bayesian side, so that's actually in 2022 I gave a talk about tie-in models and that. Tie-in models are the foundation of all of our MMM work, but we do Bayesian and for a very specific reason of where, especially in the Lego group, we're a very seasonal company. We do look at our sales trends. They spike every Q4 and all of our marketing and every single one of our inputs basically has that same pattern. So you have an overfitting chaos anytime you even run the simplest model. So we found that we're using Bayesian as the first layer to kind of add a lot of business context into the model. What we expect to, like when you actually get to the MMM part, we have these Bayesian priors to kind of help us really keep everything within constraints and we do a lot of research before we get to the modeling process to actually look at these patterns and help them inform those priors. All right, that was a lot of stats.
So we found that we're using Bayesian as the first layer to kind of add a lot of business context into the model. What we expect to, like when you actually get to the MMM part, we have these Bayesian priors to kind of help us really keep everything within constraints and we do a lot of research before we get to the modeling process to actually look at these patterns and help them inform those priors.
Yeah, that's more than I expected. I was expecting a Lego question or like a toy question at first, so yeah. If anyone has any questions about any of those terms, put them in the chat and everybody in the chat is going to help you understand them. So I can help fill in gaps too if Ryan's talking and you have questions. Thank you so much, Dr. Ishii, for that great question and there are some other ones in here that are a little bit more technical. We can switch back and forth between some technical and some non-technical. I wanted to ask Zach's question though because I love it so much as a lead-in to some of the like the silly project stuff. So Zach had asked, what was your favorite silly project that you did?
The brickr project and working out loud
Which one? I have a favorite Ryan project. Yeah and it definitely shifted. I think if I look back at everything I've done, brickr was the one where I turn images into Lego mosaics and I did that way before I joined the Lego group. That was my favorite just because that one was just one morning. I just wanted to see if it was possible and then it evolved over time and I was sharing on Twitter a lot and like I got a lot of community feedback and that was the one that I felt like my closest to my identity. I kept on iterating on it, learning more things, eventually turned into 3D models based off of Tyler Morgan Wall's work and then miraculously the Lego group was hiring data scientists in my state and that was an amazing interview topic of conversation. So I have a very, very like warm spot in my heart for the brickr side project.
Do you want to say a little bit about what it is? I put a link to it in the chat but for anyone who hasn't seen it, what is brickr? Yeah exactly. So it was it started out as so the Lego group had an online tool to make mosaics of Lego bricks from images. You'd upload an image and make a black and white mosaic from some a predefined selection of bricks. It's a very actually pretty simple math problem so I wanted to see if I could do an R and that's kind of when I I hadn't used dplyr a lot by then but I was trying to learn it more out of or do more dplyr and so I wanted to see if this was possible with the tidyverse and it definitely was and then because it's all digital you can have any constraints you want or you don't have all the constraints on the bricks so you can make colors you can make different size bricks and just basically literally turning an image of your face into Lego bricks and then over time that blew up to a lot more 3D models to instructions to like tell it but mostly the square bricks but you could say I wanted a 3D house you draw an outline of it with some rules and then brickr could create that it as a 3D model in R that you could spin around and play with so purely for fun but it was a lot of fun to build and it was a lot of fun to use and this was kind of the days when twitter data science and twitter is huge and you could just share stuff in the process and get a lot of input and everyone's cheering you on and definitely miss those days but it was it was a really fun time to develop things so it's definitely my favorite side project.
I do have some yeah we're bringing those days back we the the scene on blue sky the rstat scene and the data bs scene is really really good I would recommend it if you aren't on blue sky everybody go um go check it out go see if you can rekindle some of your rstats twitter and then also we have the posit discord which is just a data science community discord where you can do all of that you could share your stuff and it will only be going to other data people who will help you you know give you feedback or cheer you on and I will be putting a link to join the discord in just a second. Um blue sky groups there are some feeds you can follow in blue sky and there's also starter packs in blue sky if someone in the chat will help me out by putting some starter packs um Jeremy Allen has a good one for rstat people that would be great.
Um but yeah I wanted to call out also that um Eric had put his shiny app his lego mosaic shiny app in the in the chat which is a really great example it's like a mosaic maker of building off of the work of somebody else and I would love it if you go investigate that. But Ryan I wanted to talk just like two seconds more about you built this thing for lego and then lego happened to be looking for a data person and your project happened to be the right thing. I talk all the time about working out loud and the value of it. If lego had been hiring and you hadn't done that you hadn't put the time in to make this project do you think you would have gotten the job do you think it like helped you put you over the edge?
So um lego wasn't looking for mosaic or my brickr mosaic thing um I think it um but it was just a very good topic of conversation in my interviews. Actually my first interview was fine um it was like 8 a.m after being unemployed for three months and I wasn't really in the mindset and it was a fine interview but it wasn't amazing. And then in between that first interview and then the ambiguous next steps of a second interview I think some people who were already at the lego group heard that I was interviewing and actually uh one of them was very active on twitter at the time and she's like oh she told my boss at the time oh have you seen this and that got my my second interview I think. So I think it recovered my awkward first interview and him know my boss knowing about this project I did on the side that like really showed off my skills better than I could just show off an in a first round interview I think that helped pique their interest a lot more. And so on the second round interview I talked a lot more about the development of this and I was a lot more comfortable because I was in control of that conversation and I could talk very clearly about what my goal was and it showed my passion for the brand so I think yeah definitely it helped. And um I was shy about it at first I should not have been shy about but I just kind of assumed that a professional person wouldn't want to see my silly toy project but I was wrong and yeah now when I'm interviewing people I love seeing these side projects.
I was shy about it at first I should not have been shy about but I just kind of assumed that a professional person wouldn't want to see my silly toy project but I was wrong.
Um so yeah I made a miscalculation but the universe kind of helped recover me um but yeah it was definitely yeah it definitely helped get me that job I'm 100% sure of that. Yeah I'll work out loud even the silly stuff especially the silly stuff it's a better conversation starter. I have gotten two interviews because I put that I played roller derby on my application or on my um it was on my resume it's the bottom of my resume like I play roller derby that's the fun thing about me and twice I had interviews where I got the job where they were like we literally only just wanted to talk about roller derby like we we saw that you played it and like we thought you're like the fun person we put in our stack and I ended up being the best candidate maybe I wasn't very good at writing my resume to show that I was the best candidate but I was good at showing I was fun.
Scoping side projects
Okay let's head back over to Slido thank you for everybody um chiming in in the chat with all of your your fun stuff here. I want to have Noor unmute and ask her question live if she's available Noor are you there Rachel's going to help you unmute.
Hello uh uh I grew up on legos like literally my dad gave me like the not like the big legos but like the tiny mechanical like tool legos and was like knock yourself out I was five. Um did that have an input on my life maybe. Uh but the actual question is uh you talk a lot about like doing known you're known for doing like fun side projects and so forth how do you scope your projects or do you even like have a concept of scope when you go in how do you know when you're done or what do you define is done and do you know this before you start or you're just kind of like this is cool and then eventually you wrap a project around it?
I think you will see by my lack of CRAN and my lack of or my big uh my abundance of incomplete repos that I don't scope out very well. Um I basically have a random idea while watching walking the dog or watching tv or seeing what someone else did get super excited and obsessed over over for a few days or a few weeks depending on how big the thing is I definitely do not plan.
this stuff out well um and then I just build it and build it until I see can I do it and then if it's a cool thing to share I put it out show my friends or show the internet or show my co-workers
sometimes a lot of times I get bored before I have a complete thing to do but I've still learned something from it um but yeah in general I'm not a very good planner I love having a product manager at work to help do all that stuff I like proof of concepting things and moving really quickly and seeing what's possible um but once it gets to productionization I start and then you have to like worry about all the details and make it perfect to make sure it works out all over the place I struggle there um which is really why I like work on a data science team where there's people who love that part um so um yes my products die a lot and some of them never make it past the initial like untitled R script but it's yeah it's so fun to do almost everything I do is untitled R script in the wrong project I'm like I'm really really impulsive I'm in some other project and I'm like oh idea or someone has texted me and they're like help right and then whatever project I'm in I open an untitled R file and write them a demo code and then teach them and then I go what do I do with this code now should I put it somewhere I need like a demo repo where I can save these things yeah I save mine to a one-offs folder in the yeah the project I'm in and I forget where it is and then years later I have to do like a find and files remember some random lines I wrote but yeah yep exactly all right well Newer thank you so much for asking that question
and I had another one from Lauren in the chat Lauren if you are available to ask your question live and there might be more than one Lauren here I'm not sure um I'm hoping it's Lauren Price was it you it was okay this is an unmuted Lauren Price there you go am I okay good you can hear me now hello um I um as I wrote in the side oh I am uh just getting started in my uh journey towards Python after lots of years of R and even Stata um and the idea of not having a crayon to manage packages and not having like a single specific sort of industry standard GUI interface to use um I recognize those are both barriers to me getting started and I would like to know um either in the chat from people who are active in Python and R um or from the you know the hosts and guests um what are you all recommend or un-recommend as the case may be um and is there any like big fundamental difference that I need to keep in mind as I start this journey just in setting up my environment and um that's that's all she wrote thanks Lauren definitely use the chat
I'm a terrible person to answer this I do know right before I gave my talk this year is Julia Silgi gave a talk about getting unstuck with Python watch that for sure um we can link that way yes I bet Isabella will be able to help us go find it I'm sure she's like already on it but also your talk mentioned part of this your talk was like I'm not a Python programmer but I guess I am now because I'm now like launching stuff in Python Ryan's talk this year everybody was learning from the pitfalls of using LLMs for coding um it was really interesting please go watch it for sure I'm going to link it in the chat all right take it away Ryan
any words of wisdom for getting started with Python um try to stay as engaged as possible like again I use the LLMs for everything but it's a trap and um I rely on our platform and our engineer that work to help set up the environments because version controlling still shocks me and I don't understand that or not the version but like the packages and the packaging managers um I just have to pray that Databricks environment is set up correctly every time I start doing something and for the most part it usually is um but again being part of a I'm using I haven't used Python in a personal manner or side project manner except for uh advent of code I forced myself to do that in Python but then I give up after my day by 19 gets too hard um but at work I'm just very glad I have engineers and a platform team to help do the hard part for me so I'm very much like I hit an error if the AI can't help me solve the error I just shove it off and pray someone else can help me um I guess I'm privileged there but yeah I I am not your Python expert at all I think the main takeaway is that you are surrounded by people who will be nice to you when you are confused by Python stuff right so I would say the discord is actually a really great place to go there's a Python channel and a lot of folks there will be willing to help you I know that Daniel Chen will hop in and help too um my recommendation is Positron and UV so get Positron installed get UV set up and you will have far fewer problems than you would have had a couple of years ago trying to do Python without those things as a as a data person who's used to R um I see Megan Harris like nodding vigorously at that combination so she she could probably chime in in the chat as well and maybe in the discord server if we can convince her to become a discord person I'm working on it Megan okay maybe I'll add just for the people listening to the recording because I put it in the chat but Posit package manager also supports both Python so I put the link there and there's also a free public Posit package manager too just wanted to add that
When to use (and not use) LLMs
excellent excellent add thank you all right well I just saw a question pop up um immediately in the slide after you said that it's a great follow-up um it's anonymous so I will ask it it says what when do you recommend not using LLMs and why when is it a trap
yeah and again I'm not an expert in LLMs I am using them I'm trying to figure out how to use them just like I think everyone else is um I still want to have fun doing data science and I don't want to offload all that fun to an LLM so I'm still a little stubborn with if I know how to do something and I like trying to or I I know I could figure out how to do something I like trying to figure out myself and then maybe using the LLM to help me with like small discrete unsuck things um they're great at speed they're great at when I'm feeling lazy like sometimes when you're doing tons of data processing and you know you have this 40 line pipeline you're gonna have to write in dplyr be like I know I can do this let's just have the LLM do it um but if I'm trying to expand my knowledge and you're like really get close to the data and or try to learn something new I'm not relying on the LLM that much um see I feel like I'm still writing a lot of manual code especially if I'm writing stuff in R I'm still doing 99% of it myself um the Python stuff though it's just I could be staring at a blank page for a very long time until and then or I could describe exactly to the LLM what I want and then I try to fix it but yeah um I don't know I still like having fun writing code so I write more than I probably should but I don't know I'm still learning how to use LLMs best for my work and I think we all are and especially as the LLMs get better I'm sure I'll start shifting it more and more we all are absolutely and I think for me the trap comes when I go into it thinking this will save me time because it never does
the trap comes when I go into it thinking this will save me time because it never does
and what I should have done from the beginning was use the LLM to help me build a foundational understanding that I need to build the thing myself um or the foundational understanding that I need to check its work when I ask it to build functions or bits and pieces for me so that's the trap for me and now I'm going to hear Admiral Akbar in my head every time I open chat GPT thank you Ryan okay there's a question from Luke that I thought was really great Luke are you able to unmute and ask your question live wait let me yeah sure hey there he is thank you yeah so I mean I admire actually a lot of people in the hangout who are able to put out like newsletters or podcasts or even just like for you like projects and just throw them out openly in the public and be like here it is it's not perfect you know hey enjoy like I don't for some reason crossing that idea of hitting publish on something publicly that is not for some reason like groundbreaking or perfect is just like impossible for me so kind of how do you shut up your own inner critic and just make it happen what a good question thank you for asking
Sharing imperfect work and working out loud
Luke I yeah I don't think I actually have shut it up I'm almost always second guessing stuff I put out there and like back when I was on Twitter a lot I had deleted so many things not even just data science for that I'm like oh this is a good idea and like no never mind this doesn't need to be said but then yeah like sometimes I just get really excited and my family isn't can't they're not nerds they can't show their excitement I just need to find like the correct audience for even I know no it's not perfect but like data scientists are all they love imperfection they love incomplete things a lot of my side projects were inspired from other people putting things out there even putting incomplete things out there and me seeing what I could do with it and spin it off so I guess that this when it comes to sharing incomplete stuff I don't think about it at all I just like that's the natural thing you do you're creating things and I know I'm never going to perfect it so it's either hide it away and or just put out what you have there and it might inspire someone else or someone might learn from it and clear I'm not doing as much as I used to do but I wish I would do more but I'm hoping this kind of talk today inspires me to keep doing more and more of it well yeah I don't think I ever made an active decision to shut myself up from or like be afraid of not being afraid of releasing stuff I think it just came naturally seeing everyone else around me doing it and there is no risk to putting out silly ideas out there if they're just fun side products of you learning I agree but I do recognize
that it's terrifying I will say I'm going to put the link to my blog in the chat and I would like you if you care to go look at it and see that I literally published a like seven to eight part blog series on something that no one cares about like I wrote an app and spent a year learning deeply something that absolutely nobody cares about and it's not perfect but I had so much fun the takeaway though is that no one looked at it I just published it and I was like it went out into the internet nobody said anything mean to me about it because nobody knew it existed I think I let a few people know on blue sky that it existed but again it's like a topic nobody cares about so why not I published it but whenever I needed an example of like working out loud or you know my process because it demonstrates my process I have it all there people can go look at it far fewer people are looking at what you publish on the internet than you think I guess is what I'm trying to say unless you really really try to let them know it's happening like you give a conference talk like Ryan did okay let's head back over to Slido
thank you everybody for chiming in in the chat with all of your plugs and your links and stuff oh and I will also plug Simone Yetch as a person who intentionally decided to do bad projects out loud she's the girl who like builds bad robots and she has her for many many years you can go look her up on YouTube fantastic inspiration for working out loud okay we had a question from
Natasha and if this question is one that Ryan cannot ask or answer because it's specific to Lego I want to say that that's okay but Natasha if you're able to ask that live I would love it hi so what I was wondering about is what kind of data you use for your LLMs and stuff and just I can see that with something like Legoline from the outside we're all seeing that oh everybody's excited on Twitter or at least all the social media places you would see a lot of excitement about new Lego sets but like for your purposes where do you get your data from
Data sources for marketing mix modeling
yeah I think in general terms whenever you're buying a lot of marketing and investing behind it you get a lot of data about that so if you're buying from Google or you're buying from Meta they give you a ton of impressions data you know exactly what type of audiences are seeing your ads who's seeing it who's clicking it so basically where we if it's marketing traditionally where we buy it we get the data from we have to account for a lot more things than just digital marketing because especially with the Lego group it's a brand that's been around for 95 years and has a lot of brand health and a lot of affinity and then I think the interesting thing is as you mentioned there's this very niche group of Lego nerds and it's very highly correlated data science nerds and they know everything's happening with the Lego group all the time and we don't really have to market to them we just have to get them excited for it but there's a whole other larger circle of people who don't really know much about our product so they think oh it's just a it's a kid's toy that you can build fire engines with and we have to do a lot of marketing and a lot of things to get them excited for and social media is great for that uh we have a lot of data in our models it covers everything covers different audiences uh but I guess from we can't get it from we get a lot of different sources so um it's hard to answer that question and sound bites for general audience but yeah happy to talk about this stuff challenging well why don't we hop over to another sort of
modeling question because Mark had asked um one of those Mark if you are available I think this is Mark Ricky because it's an MMM question and I know Mark maybe he can try to see if Mark maybe has first initial name or if he had to drop I think he had to drop 30 minutes and you might have had to draw let me ask his question um because it got upvoted right by a few people um and it was does Lego use an open source MMM package like Meridian or Robin or do you like build from scratch use an external vendor he was curious about how that decision process might have been made
yeah that's first of all I'd say I was not expecting this much excitement about marketing data science or MMM so I feel like maybe I need might need to join discord and we set up a marketing nerd uh chat or something because I think there's a lot I want to learn from the industry as well
um but to answer the question directly uh yeah when I joined we had some Robin wasn't out there yet because it was 2019 but uh we had we have external firms and agencies that run MMM for a lot of our markets but they're very expensive and so for and eat a lot of some pieces of the marketing budget so I built one internally or we had to build run some MMM models internally uh my team and I chose to develop our own um and I'm on the second iteration of that right now and again I use tidymodels as the foundation of all that uh was really what I needed at the time to have all the control over the inputs I built a lot of wrappers around that um so we're using an internally developed package in R right now um and I love it and I it works very well for us
and so I I like maintaining that um it is always discussion what the next generation that's going to be um but we have so much in our pipeline right now that I think my package is safe for a few more years uh it just works very very well we can control everything we need to control for it um so yeah I'm very proud of it so and yeah I think I talked about this a little bit in the 2023 talk that was about the development process of that MMM
yeah I was actually about to put that in the chat one more time so I'm going to pop um the link to his 2023 talk in there all about model pipelines and I know that you have said like you use tidymodels all the time but you don't actually ever touch tidymodels because you wrote things that just incorporate it behind the scenes um I don't know if you wanted to just talk really briefly about about that topic
yeah so we I had to write a bunch of custom step if you've seen tidymodels there's a lot of process like it's basically setting up your model and then you're running it independently your recipe and you're running independently and there's step functions I had to write a lot of custom step functions to fit our data transformations that we needed to do for MMM uh but the biggest challenge we have with MMM um besides the priors and running the models and getting all your hyperparameters is that we are at every time we have thousands of data points that we could or data series that we could put into models um a lot of them are highly correlated a lot of them are different ways to measure the same thing so if you have marketing you can see you can see your investment or you can see your impressions which are one for one correlated you can't use both so we need a lot of ways for the data scientists and the business expert to know which ones to turn on and which ones to turn off so I built a bunch of wrappers that make it very easy to kind of set your transformations turn your toggle your features say what transformations they should have because you need some s curves for if it's uh if you're expecting diminishing turns and decay sometimes you just need some c curves again this is we should have a marketing hangout I think um we totally should but yeah um so yeah I forgot the question point um but yeah that's okay you did it. Tidy Models is great yeah I built wrappers for it so I haven't actually touched Tidy Models itself in a few years just because we have a bunch of wrappers running in the background but everything I like every Tidy Models step the recipe the baking the um the running um that happens every day in our modeling.
Communicating MMM to stakeholders
I love all of these questions and Ryan if you did not think that you were going to get modeling questions you would be shocked to look at this one right now because I'm definitely surprised yeah they're all they're all in there um so why don't we do at least one more we have a kind of a handful of marketing questions um let's see what could help us bring these all together one of the questions is about communicating to stakeholders so it is about MMM but it's a it's a question that I think can be broader than that about stakeholder communication it's how do you communicate the nuances of MMM like priors um Bayesian versus frequentist whatever to stakeholders who may not be technical or do you even communicate any of those nuances do you stay higher level than that?
Yeah so I have a wide variety of stakeholders with a wide variety of interest in the details and how much information they want to know about the back end and I have some stakeholders who I've only worked with for a few months some who I've been working with for five years and who I'm more comfortable sharing more and more details to without overwhelming them um I don't think any of them have had asked the Bayesian versus frequentist question because that's a very very niche question to people who are in statistics every day um but in general we have to explain to them that every kind of variable has a lot of different transformations we need to apply a bunch of assumptions about carryover impacts and diminishing returns and the size of the impact and everyone seemed to get those very high level terms so we have to basically have a bunch of inner mappings in my brain or on slides to link these very complicated data science transformations into sentences that they might resonate with the stakeholders.
Everyone's also like always very excited to hear about how we're using their data in our models and trying to get as much information at this as they can. I think MMM is tough because since we're trying to cover every single data point we can't really drill into every super specific question they have at all times so it's about training them about the best way to use the model results and then once we deliver our results how they can integrate that with their multi-touch attribution their last touch attribution or their other attribution models. In general I love talking to stakeholders and sharing as much as I can about them and it's really cool to see them get excited um I've just I've learned that when I first started they'd ask they'd ask like technical questions I'd go all in on the very technical details I'd lose them right away so I now have to I have methodology slides and like different detailed methodology slides or presentations I can give them and I try the easy one first and if they're all on board and then I can have another conversation with them going another layer down but usually what I find is I think of the high level ones they're usually good enough for most people they want to feel like they understand the process um but yeah math and statistics is complicated and so uh it's very easy to lose them if they tell them too much information.
I'd go all in on the very technical details I'd lose them right away so I now have to I have methodology slides and like different detailed methodology slides or presentations I can give them and I try the easy one first and if they're all on board and then I can have another conversation with them going another layer down but usually what I find is I think of the high level ones they're usually good enough for most people they want to feel like they understand the process.
Does a love of Lego help with coding?
It is I love it okay let's hop over to the slide one more time and see how many questions we can get in our last 15 minutes there was an anonymous one that got asked that I absolutely loved um and it was does your love for lego help you build code better like scoping sectioning loops modularization I guess is is one that I would throw in there because like it was great about that I haven't put those two things together but I I think there's probably a real thing there um again understanding how pieces fit together and understanding how to link different groups and splitting up correctly I I think I could see that um slightly unrelated I think my love for lego helped me do my job better in general just having being extra excited about the dad that we're using so if you don't care about the brand or so I reversed my old job I did a lot of market sizing and forecasting for tech companies I was doing some similar work not mmm but a lot of modeling and our code and I did not care about the domain at all um it's about like networking pieces and um cpus and just shipments of computer components I wasn't passionate about the domain so I think I was not passionate about the work and then once I got the job with my current company I was doing data science which I love doing and I was working with data that I was actually excited about um I mean if you know our portfolio we do star wars products we have jurassic park products I get to work with data every day that is all marvel pop culture and really fun data so um having that enjoyment with what I'm building really motivates me to do to enjoy my job more and do better at that
thank you for that fantastic question um whoever you were anonymous asker and it also reminded me and it reminded Alyssa to the educators in the chat are talking about the same thing um scratch if you've never played with scratch it's very lego-esque it's basically teaching kids to code and it puts everything together in these like interlocking pieces and it's visual right so it's a it's a ui and you're putting together these little pieces that snap together and do different things right like different types of what we would call functions um that operate on each other and um it's super super fun if you've never played with it I highly recommend it go create some things in scratch but it's very very modular and lego-ish
Team priorities and workload
okay let's hop back over to Slido bear with me for one second while I read through here um I saw a question from Tim Tim Golden if you're available to unmute and ask that live that would be fantastic sure yeah what what was I going to oh yeah right priority priority for sure yeah uh how do you decide what your team works on next so what what are the things that drive priority for for what your team works on at a company that we're not diluting the brand of
yeah um short answer is I personally don't decide too much um we and this has shifted a lot over the past few years but we have I sit within a marketing portfolio team so everything that team cares about is optimizing marketing and then it trickles down to my product which is all about marketing effectiveness or marketing optimization so we're trying to we have a lot of people that are so we're trying to we have a lot of demand for our work but the big overarching thing is what our team's goal is and that's to build reproducible marketing optimization tools and then trying to fit within that um so yeah I have a product manager now we have a portfolio manager so um everything most of those decisions and choosing what to work on really have to make sure what's the most aligned with our product what can we think we can reasonably do within that certain amount of time um what one will have the most impact uh business impact at business value so there's a lot of really cool questions we're asked sometimes and I would love to answer them but it's really hard to fit into schedule and it's it's hard to really understand to uh justify like doing this small project for the small team that might be super interesting but it would come at a risk of losing our some of our marketing optimizations um but that's kind of just the part of being part of a big company and having a lot of different teams doing data science we have some like incubation opportunities and hackathon opportunities to kind of prove extra value and those are always fun um but yeah we're not short of work so it's always we have to rely on the business to help prioritize fantastic thank you so much Tim for asking that
Resources for marketing analytics
all right in our last 10 minutes here I'd love to do rapid fire if I can so questions that might be able to have shorter answers um and you can go as long or as short as you want with them but let's see the first one is resources for learning marketing analytics this wraps in David Gerhart's question from Slido which is have you learned or applied anything from Rory Sutherland who is a speaker who talks about marketing all the time um and if there are any books podcasts anything that you like that we can find the links to and share in the chat with everybody
a short answer I don't know um I struggled when I first joined and I had a catch-up and I read a lot of stuff and a lot of stuff I read wasn't super helpful um again there are some people I follow on LinkedIn that have this mmm adjacent stuff and they give talks I I don't I read this stuff but yeah I don't have a good answer sorry for that I think that's a really great answer actually this you're like I read some stuff and it wasn't super helpful is actually a nice navigation tool because there's probably a lot of people out there going maybe I should be reading all of this stuff and like you know not all I don't think I have every single person either but yeah I bought a thick book about marketing data science and R when I first joined my job and none of it was what I was applicable to my job or anything I was very disappointed by it so I actually on my to-do list it's like I'll start writing books or blog posts about marketing data science but that's a lot of work it is a lot of work to be the change you want to see but also be the free resource that you wish you had it's it's a full-time job
all right next rapid fire what tools for Bayesian analysis do you use do you stan and BRMS yep so tiny models works very well with our stan arm before that I used stan which I enjoyed the extra low level of flexibility it gave me but when it came to like define the
models and forcing some coefficients to behave more than our stan arm lets you but I kind of liked how our stan arm has a philosophical restrictions on what transform or distributions you can use so because their thought is if they don't support it then you're kind of cheating with some of your distributions like you're truncated normal you could force everything to be positive or negative you want but then you're just kind of you're losing confidence in the model results so our stan arm for sure through tidy models and I've learned to love it slowly thank you for asking that question whoever you were
Career path and lucky accidents
and we had one that was possibly quick to answer it was was your move from tech companies to the industry where you care about the context intentional or was it a fortunate accident
I'm the luckiest person literally my old company closed down and the week after the lego group in my home or my home state hosted two data science jobs and yeah I just got lucky I can't describe it anywhere else so I ran with it I yeah I had some other job offers in in the process and I like lego was slower than the other ones and like I almost took a job at another tech company and I was debating it and obviously tech companies pay more but then I remember just like go with my gut go with the lego group I'm so happy I did that but yeah I'm so lucky I don't take that for granted I wish I remembered what ariel drawer called it or someone someone asked a question to ariel drawer the other week that was like is there a passion tax or something or like a passion penalty for like doing what you love and getting paid slightly less but not all compensation is monetary either it's a nuanced thing and that other company I turned down has had so many layoffs since and like who knows what universe I'd be in right now it's like I mean even if I the salary is lower I think quality of life is so much higher
who knows and I will say yes there was an element of luck and randomness but you had already worked really hard to build something that showed that you were a cool fit and that you had a lot of like passion about the thing so you already did the work like the hard work that you did increased your likelihood that the luck could land on you right and I think that that's important to to call out that it was it was your hard work as well
End products and reporting
okay natasha had asked one that I would love to see if we can do rapid fire and it was uh what is your end product do you summarize your results into reports and presentations or do you only handle the analysis great question Natasha yeah and that's a huge topic conversation and at first I think maybe two years ago the MMM the delivery of the model results was the end product and um I I do those presentations as well so I I work with the business to interpret the results but then I share them out to the wider business and my team and I do that um but now I think it's shifted to a lot of using those models for more things so it's all about marketing optimization recommendation scenario planning so now that we have all these coefficients and all these relationships that we know between marketing and sales do more with them we still have to do the share out but then we have to basically create apis put out those model results and those relationships into apps so we are building front end stuff so we actually have a live shiny optimization tool that my co-worker built that's being used by our marketing team
uh but we wanted to keep doing more and more of that and turn those so the MMM model is just an input to our other bigger product which helps people actually make better decisions so it used to be about describing the past and helping them learn from that but now it's actually about doing more of the models and it's a lot of really fun work which is probably why I'm still super engaged six years into the role fantastic that's another lucky thing right I've been doing this for six years I'm still finding all the fun in it
Bayesian MCMC tooling
thank you Natasha for that fantastic question um there was one that we might be able to do pretty quickly and it was what is the modeling tool that you use to implement a Bayesian MCMC model and for anybody who is sick of all of our MMMM things um MCMC is Markov Chain Monte Carlo
and so yeah I think it's just back to the RSAN arm and the uh tidy models I think it's all the work for me yeah okay fantastic well we can blow through these that's great
Automated reporting and scalable pipelines
how do you make your model-based reporting faster and more automated like how do you set up scalable pipelines that make manual tasks like this much faster I think this question is from Satorishi and I think that your talk in 2023 might be a good resource for this one is that do you think so
yeah that's part of it I'd say the rapid fire other half of it is that the marketing landscape our business is always changing or like the business the questions that the business are asking of us and of the models keep changing so I can't automate all this stuff we've tried but I think we've risked feeling stale and not answering the questions we need to answer if we automate all the output so we we do deliver pivot tables which are used by a little few people we have some power bi's but the really interesting part about digesting the results is that we hear what the big hot topics are in the business before modeling we see if our model will be able to answer that or do we have to add new variables to do that and then looking to see how those relationships are popping out of the models then building custom reports or just even if it's charts or one page slides that's a really fun part just because our business is always changing that's why I have to retrain our model so much and really seeing how those relationships change so I care a little bit less about the automated reporting we do do some but I get more excited about the questions that are coming out of the models that everyone's interested in right now.
Career advice
Nice all right I think we did pretty well in our rapid fire we have just a few minutes left and in that time I would love to ask you what we try to ask most people which is do you have a piece of career advice that you have found really helpful or that you like to give others?
I wish you had warned me about this. I know I was just thinking beforehand and I feel like I didn't warn Ryan. I again and I know some of it's luck just finding for me just being passionate about the domain I'm in has helped me so much with my career because like I'm excited to do the analysis I'm excited to do the work I just really like talking about my job and when I'm producing now so I love talking about it I love showing it and I think that passion has shown and my bosses appreciate the passion so I think that has helped me more than anything I'm not the strongest data scientist I'm definitely not the strongest ML person and as you know from my Python talks I'm not good at Python but I'm really good at just being passionate about the data knowing the tools to turn business questions into data science tools and then turn those back into the business that's what works really well for me and I would not be excited to do that if I was like in the tech industry that I left like I just do so well at a company that I really enjoy and I'm passionate about so not concrete advice but try to find that passion.
knowing the tools to turn business questions into data science tools and then turn those back into the business that's what works really well for me and I would not be excited to do that if I was like in the tech industry that I left like I just do so well at a company that I really enjoy and I'm passionate about so not concrete advice but try to find that passion.
It's still I mean it's like do a temp check on on what you enjoy and what you don't enjoy I guess is the the follow-on that I would I would have like knowing what you don't like to do can sometimes be just as valuable as knowing what you do like to do so one thing that I did was I plotted all of my jobs on an x y axis of happiness and time so like over time here are all of my jobs and then the y-axis was happiness with smiley face icons and I went through and plotted and then I went through and asked myself what made these happy jobs happy what made the sad ones sad um I recommend everybody go do that
all right everybody I have had such a good time Ryan thank you so much for joining us I hope you had a good time yeah no this was way more fun than I I was stressed but now I'm good yeah oh we made Ryan feel not stressed well we're gonna get Ryan on the discord server if you are on the discord server make your way to the data science hangout channel and tag me so I can make your name yellow give you a little badge that says dsh um and we will have to plan um marketing chat at some point so we can all have more technical questions yeah definitely and modeling it's gonna be fantastic all right next week Kevin Dalton will be joining us senior data scientist at Great American Insurance Group and then we have Thanksgiving which is off I will not see you next week Rachel will be here with you Isabella will be here with you um and I'll see you the week after Thanksgiving hopefully thank you so much for being wonderful if you want to save the chat there's three dots in the top that will let you do that it'll save it as a txt file and of course if there are any resources that we can go save over in the discord everybody go do that and eventually maybe we can have our chat only on discord and then we don't have to worry about saving chat files because everything will live there so as we move in that direction towards discord chat highly recommend going and joining the discord okay because it will be the place where we can continue conversations and we won't just have an ethereal chat that goes away every single time all right everybody I will see you in a couple of weeks but come hang out with Rachel next week with Kevin Dalton okay bye everybody