Resources
Apr 11, 2024 |
1h 1min
|
600 views

MLOps for a billion pizzas a year | Zack Fragoso @ Domino’s Pizza | Data Science Hangout

We were joined by Zack Fragoso, Head of MLOPs at Domino's Pizza to chat about his mission to Sell More Pizza and Have More Fun by making the business smarter with data.

Speaker bio: Zack Fragoso is the Head of MLOPs at Domino’s Pizza, where he is on a mission to Sell More Pizza and Have More Fun by making our business smarter with data. His team develops ML/AI applications that supercharge the global pizza delivery leader’s e-commerce and in-store platforms. We've been able to help reward customers for ordering pizza, give them a better estimate of when their order will be ready, route orders more efficiently, and those are just a few examples! Zack is passionate about MLOPs tools that increase productivity, enhancing cross-team collaboration, and support our model life cycle management.

At 11:07 we discussed two key challenges regarding MLOps: scalability and process

Scalability

Zack explains that while data scientists can develop accurate models on their laptops, scaling those models to handle the immense volume of Domino's business is a significant hurdle. He emphasizes that Domino's sells close to a billion pizzas annually, with a high concentration of orders between 5 pm and 8 pm. Events like the Super Bowl push the system even further, requiring the ability to process around 6,000 orders per minute. For Domino's, this means that models must be not only accurate but also highly performant to handle these spikes in demand.

Process

The second challenge Zack identifies is related to integrating the experimental nature of data science work with a robust and reliable production process. He acknowledges that data scientists are inherently driven to experiment with data and various modeling techniques. However, translating these experimental models into a production environment requires a more structured approach that encompasses control mechanisms, versioning, and a focus on reliability. Finding the right balance between this need for flexibility in experimentation and the demands of a structured production process is an ongoing challenge for Domino's.

Resources mentioned in the chat:
🔗LinkedIn Group for the Hangout: https://www.linkedin.com/groups/12610075/
🔗Domino's Emergency Pizza Website: https://www.dominos.com/emergency-pizza/, offering a free emergency pizza for those who order a regular pizza.
🔗Dominik Vajda shared an article from The Guardian about a Belgian brewery using AI to improve beer taste: https://www.theguardian.com/technology/2024/mar/26/ai-beer-taste-better-belgium-scientists


________________________

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software

To join future data science hangouts, add to your calendar here: https://pos.it/dsh (All are welcome! We'd love to see you!)

Thanks for hanging out with us! 💛

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi everybody, welcome back to the Data Science Hangout.

If we haven't had a chance to meet yet, I'm Rachel, I lead Customer Marketing at Posit. I'm so excited to have you joining us here today. The Hangout is our open space to hear about what's going on in the world of data across all different industries, chat about data science leadership, and connect with others who are facing similar things as you.

We get together here every Thursday at the same time, same place, so if you're watching us in the future and want to join us live, there'll be details to add it to your calendar below on YouTube, but just make sure that it adds it for 12 Eastern Time so that you can join us live. I know some people had some issues around like daylight savings time or with the time zones, so it's every Thursday from 12 to 1 Eastern Time.

I always love to ask, is this anybody's first Data Science Hangout? Say hi in the chat because we'd love to welcome you in and say hello there. We are all dedicated to keeping this a friendly and welcoming space for everyone and love to hear from you no matter your gears of experience, languages you work in, titles, or industry.

It's also totally okay if you just want to listen in and you can be a part of the Zoom chat party that you'll see happening here, but there's also three ways you can jump in and ask questions or provide your own perspective too. So first, you can raise your hand on Zoom and I'll call on you. Second, you can put questions in the Zoom chat here, and if you're in a place with a loud background or something and you want me to read it, just put a little asterisk or star, as I call it, next to it. And then third, we have the Slido link, which Curtis will share or maybe already shared, where you can ask questions anonymously too.

Before we jump in here, I have two other notes I wanted to share with everybody while we're here. So the lowest price for the POSIT conference expires tomorrow, so I just want to make sure everybody knows that. If you are thinking of joining us for the conference in August in Seattle, I encourage you to go take advantage of that before that goes up by tomorrow.

And then next, I am on a personal mission this year to help connect people who are using POSIT across their own companies. It really pains me when I find out somebody didn't know they had POSIT Connect somewhere in their company and I could have helped connect them. And so our LinkedIn group might make it easier for you to find people on your own from your companies, but I would also love to help introduce you. And so I just made like a short quick Google form if you are ever curious and want me to help connect you with other people from your company.

But with all that, thank you so much for joining us today, and I am so happy to be joined by my co-host, Zach Fragoso, head of ML Ops at Domino's Pizza. And maybe we kick off the party in the chat with sharing what's your favorite kind of pizza. But Zach, I'd love to have you introduce yourself, share a little bit about your role and something you like to do outside of work, too.

Sure. And all of them, I think, is an appropriate answer, too, if you want to drop it in there. Hey, everyone. Thank you for having me today, Rachel. Thanks for the invite. I've been really looking forward to this. I've participated in a few of these, and it's really cool to be on the other side of it presenting. So thanks for the opportunity.

My name is Zach Fragoso. I'm coming to you here from sunny Ann Arbor, Michigan today. And I'm kind of here to talk about the Domino's data journey that we've been on, and also as I was writing some notes for my talk, I thought, well, my journey is kind of weaved in there, too. So I'd be happy to talk about my career trajectory and opportunities, roadblocks I've faced along the way as well.

But just to get started with a quick intro, I'm the head of ML Ops at Domino's. So in that capacity, my charter is to make sure that we have the right technology and process available to our data science team to be able to deploy our models into our production environments and integrate them with whatever downstream platform we're integrating with, either our e-commerce platform or in-store technology platform.

My team also covers global store siting. So using geospatial data and machine learning models to identify the best places to build new Domino's stores and supply chain optimization as well. So looking across our network of stores and supply chain centers and finding ways to optimize that network to save us costs, but to also help the environment, too, which I'm really excited about that impact as well.

I play guitar and I just saw Dune II last night. I finished the book over the weekend and frantically, I wanted to finish it before it got out of theaters. And I literally went to the last IMAX showing of Dune II last night at 9 p.m. So I was up until like midnight, the last IMAX before Godzilla takes over the IMAX theater near me.

Nice. That's awesome. I'm working from my parents' house because I'm going to be here for Easter. And so I think my dad was about to make an appearance on the data science hangout as well. So that's why I was like waving over here. I have three kids that are home on spring break running around on the other side of that door. And I just found out the lock doesn't work, stopped working. So you might meet my kids during this as well.

The Domino's data science team

That's great. Awesome. Well, yeah, I'd love to learn a little bit more about the data science teams at Domino's. If you maybe can kind of share a little bit about how that's structured. Yeah, yeah. I've been lucky enough to be a part of a really exciting growth journey at Domino's. So I joined the Domino's data science team in 2018. And around that time, I think we had 10 to 15 data scientists on the team. As a team stands now, we have around 80 data scientists. So pretty rapid growth over the last few years, really focused on the business impact we've been able to deliver on.

But we have 80 team members and they we have some in Seattle, we have team members in Ann Arbor, we have team members in Amsterdam, and we have team members in Singapore. So we're like truly a global data science team, which is exciting. We have some really cool partnerships, one of them being with the Posit team, which we're really excited about, but also NVIDIA and Microsoft and Docker. We have teams specializing in pricing, personalization, retail technology, store siting, forecasting, supply chain optimization, and MLOps. So cover almost every area of the business. And we have about 15 machine learning and AI models in production right now serving various business use cases.

It's amazing to hear like how the team has grown to 80 people. What are some of the, I guess, takeaways or lessons learned from you and growing the team in these past few years? Wow. So many. Is that diving like too deep right from the beginning? Yeah. So many. I think if I were to give one piece of advice, you know, for someone kind of going through this journey or about to embark on this journey, it's really everything starts with the data. I mean, it's so cliche now, but it's truly garbage in, garbage out. So focusing on having really high quality data that's not only exists, but is really in an accessible format for data scientists to be able to leverage to kind of extract that business value. I mean, any team that's starting, that's got to be the starting point.

it's really everything starts with the data. I mean, it's so cliche now, but it's truly garbage in, garbage out.

MLOps challenges at Domino's

I see quite a few questions coming in now in the chat. And Arsenis, do you want to kick us off here with your question? Yeah. So just, you know, thanks, Rachel. So very, very quickly, just really high level to start out, I was wondering what some of the more unique challenges are in MLOps that Domino's faces. And also thank you, Zach, for joining today and taking the time for this, but yeah, that's my question. Thanks.

Of course. I'm going to touch on two challenges. The first one is scalability, right? I think when you talk about ML and AI and MLOps, there's a lot that can be done on a data scientist laptop and, you know, great accuracy and recall can be achieved on models on an individual's laptop, but there's a lot of challenges that arise when trying to scale that, you know, to a large system. So Domino's, maybe, probably not a shocker to anybody, but we sell a lot of pizza, you know, we sell close to a billion pizzas every year across the globe. So everything that we do has to be looked through the lens of scale.

You know, there's that volume scale, but also like our business is highly concentrated, right? So if you look, you know, at the pizza business, 80% of our orders are between 5 and 7, 5 and 8 p.m. And then that's on a daily level, but even if you look across the year, days like Halloween, Superbowl, New Year's, you know, represent a disproportional amount of our business. So we have to scale to be able to serve, you know, something like 6,000 orders per minute during Superbowl. So if we're doing machine learning that is kind of at the order level, our models not only need to be accurate, but performant to that scale. So that's one of the biggest challenges we face.

The other one is process, right? Data scientists are experimenters at heart, right? They want to take the time to experiment with the data, experiment with different modeling types, and, you know, fitting that into a process that balances the flexibility and creativity with the control and versioning and reliability, finding the right balances is tough from a process standpoint. And that's a journey that we're still going on, to be honest.

Can I add on to this question and ask, how do you address some of those challenges today? Well, many, how we address it kind of depends on the use case in terms of scalability. But, you know, we, maybe one example there is like looking at different model types and the balance between accuracy and performance. So a good example that I always go to is like, like random forests are really great at achieving accuracy, but, you know, traditionally like slow on the inferencing side. So we might make the decision to do like a deep neural net even, that might be a little bit overkill to achieve the accuracy that we need, but it's more performant from a scalability perspective. So understanding those trade-offs and having our data scientists kind of keeping that in mind as they're doing their model development experimentation is, I think, important.

Ralph, I see you had a question, want to jump in here next? Yeah. Zach, first things are being on here. I would probably have like a zillion questions on supply chains, but one's more broadly applicable. Sure. If you're kind of doing any sort of like very, very short-term demand forecasting, I've heard it called now, Cassie, that's what I call it, like next hour or so to try to level load demand between stores. Because like I've had a time where I've ordered it from a Domino's store nearby and they say, yeah, it's going to be two hours. And I look at the one that's a mile farther away, they're like, oh, we'll get it to you in 15 minutes. Like, it really mattered to me whether it's the closest or not. Like, yeah. I'm curious if you're doing anything like that to try to kind of improve the customer experience.

Yeah, for sure. I think those are actually the areas of opportunity that we really look for because they're oftentimes a win-win. Like, when we can get the customer a hotter, fresher pizza, it's a better experience for the customer. But we can also potentially reduce the miles that we're driving on the road, improve our labor utilization rates. So like, those are the kinds of efficiencies that we're kind of really after is those win-win from a customer and cost perspective.

So, yeah, we look at that from several different angles. One of them is just the store siting angle. It's where do we put our stores relative to our customers to optimize our network? Domino's, we have around 6,800 stores in the United States, and that covers around 80-ish percent of the U.S. population lives in a Domino's delivery area. So we look at our stores, not as single units, but as a network of stores and how do we optimize that network to better serve our customers? So that's like at the macro level, but we're also looking at it from even a delivery routing perspective. So what orders should be grouped together on routes to better service the customers

at the most micro level? Great. Thank you. Thank you. Let's see. Quite a few questions here, a few that have a star there for me to read it. So I'll ask Sam this question.

Zach, has your team influenced the data collection side of things, maybe to improve data quality or to start collecting new data that previously wasn't collected?

Yeah, certainly. And that is often step number one when we're embarking on a new data science or AI ML kind of project is data contract negotiation. So it kind of starts with that feature engineering, data exploration, initial model exploration phase and our data scientists kind of look at, hey, what are the most parsimonious set of features that we need to achieve the model accuracy that we're looking for?

And going through that experimentation, having other data scientists review that, making sure that we're not asking for too much. Then immediately after that, and phase number two is, all right, taking that to our business partners. And that includes our enterprise data warehouse team that includes like our e-commerce team or in-store platform team and saying, hey, this is the model we want to build and this is the data that we're going to need to kind of service that model at inference. And then it becomes a negotiation. And we literally call it like data contract negotiation between the partners. So and there's gives and takes, right? Because every data, especially when you're talking about real-time inferencing, right? Passing data, there's a real cost to it. So taking those things into consideration, balancing that with what's needed to achieve a certain level of accuracy is a big part of the initial planning for any project we have.

Community engagement and local meetups

Thank you. I wanted to make sure we get a minute to talk about the local meetup that you want to start as well. Because I've been thinking about, and I had a note to mention this, but like the Boston Usar group is pretty much sponsored by like having Domino's pizza and thinking of how many meetups around the world and how many data scientists get together and like how often Domino's is involved in that. And I know you wanted to start a group too. So just had to take a few minutes to chat a little bit about that.

Yeah, first of all, like if it's not already obvious, I've totally drank the Domino's Kool-Aid. So just like bear with me, but you know, that is like the cool thing about working for Domino's is like at the service level, it's just a pizza company. But when you realize like the magic of pizza and it's like what you have at your kid's birthday party, it's what you have at a data science users meetup, it's, you know, the night at home watching a movie, you know, with your significant other, it's those like kind of cool moments that I think are what make Domino's special. So that's my obligatory PR statement, or not so obligatory actually.

So yeah, I think in that spirit, like this is me doing this as kind of a step one for something that our data science leadership team has been talking about is like how do we engage the data science community and give back to the community and like the spirit of, you know, rising seas raise all ships and like working together and building collaborations outside Domino's. So we're looking at ways to do that, you know, one of them is, you know, hosting our own user group in the Ann Arbor area. So if any of you guys are interested, please reach out, we're in the very early stages of potentially planning something like that. So gauging interest would be would be really helpful. Yeah, if anybody's in the area, Ann Arbor, comment in the chat. I'd also be happy to help connect with people that I know in the area too.

Project ownership and prioritization

I see Alan, you asked a question in the chat, want to jump in here next? Yeah, sure. Hi, Zack. Hi, everybody. I think Zack, your answer to Sam's question really overlapped with a lot of the stuff I was interested in about the partnership and how you may have to influence like stakeholders to get to consensus about what to do.

I'll reframe it a tiny bit and say I'm curious, like as you get requests or projects, is there like a product champion or a product specialist or something like, is there someone who kind of owns it from end to end, who's really responsible for conceptualization? Or do you get like work from all over the place, which means you've got like 15 funnels kind of coming towards you that you've got to prioritize and manage and stuff?

Yeah, I can sense like where that question is coming from and I just want to know, like I feel you at the same wavelength. So yeah, that is a challenge and I don't know, maybe for, at the risk of giving you an unanswered, I think it's like really striking that balance. One of the things that really I love about Domino's culture is the entrepreneur, like entrepreneurial nature of it, right? So the fact that ideas can come from anywhere, any department, any level, and like those ideas have a real shot at making it into like a product that, or a project that, you know, gets to be seen by the world if it's a marketing window or a piece of technology that gets to be experienced by hundreds of millions of Domino's customers all over the globe. So that's really exciting.

But the double-edged sword to that is, you know, it can be kind of chaotic and, you know, keeping the reins in and prioritization then become really paramount. So having a process for taking in those really great ideas, prioritizing them, and then to your point, assigning a champion or a product owner that can be kind of responsible for that. And, you know, one of the ways we're kind of addressing that now, which is really we're trying it out, is taking more of that product owner approach, like borrowing from Agile and really having product data science, product owners that aren't necessarily data scientists or have expertise in AI or ML, but are really good at project management and seeing a project through from end to end and can supplement, you know, that skill set with our data scientists who are, you know, have that creativity and technical orientation. So that's kind of an area that we're exploring to address, you know, this topic you're bringing up.

Super. Thank you. Yeah, sounds like familiar dilemmas. And it sounds like you've got the organization sounds like it's really well positioned to have support to figure that stuff out, which is really cool. Yep. Thanks. You know, sometimes you experiment with models. Sometimes you got to experiment with process, too. So I think that we're open to that as well.

Supply chain analytics and vehicle routing

I see Arthi had a question a little bit earlier. Do you have data science models used for supply chain analytics, like the quantity of ingredients needed by a store and how to make sure they don't run out, especially on high demand days? Yeah. So really, when I think about or at least the level of technicality I can speak to on this topic, I think about kind of two different flavors of model that we implement to optimize our supply chain network.

I think one is forecasting. Right. That's that's kind of key to all of this. When you're talking about optimizing supply chain, it's not because we want to you know, we can't fix anything that already happened. So we want to optimize into the future. So that kind of necessitates those time series forecasting models. And we kind of have a whole laundry basket of forecasting models that we implement at the, you know, from the hourly demand signature level to the store level, to the system level, we look at forecasting.

And then really the where the the next step of that is really then combining that with more combinatorial optimization algorithms. Right. So not even like traditional AI or ML, but things like mixed integer programs, right, or vehicle routing solvers, right. And one of the cool things that we're doing in that area is, again, to address like the earlier topic of scalability, traditional like MIP or vehicle routing solvers, they're meant to solve logistics problems like overnight, because like orders come in, and then the next day, you need trucks, and they take the whole day to deliver those routes, and then they come back to this depot. Well, for us, like, we need to route delivery drivers, they're taking, you know, 1000 routes a minute when you look across our entire system. So we need to solve those vehicle routing problems really fast. So we've partnered with Nvidia to leverage their coop solver, which is that is really cool with the first GPU enabled vehicle routing solver, and it's able to solve those problems really fast and get to almost the very, very close to the ideal solution.

So we've partnered with Nvidia to leverage their coop solver, which is that is really cool with the first GPU enabled vehicle routing solver, and it's able to solve those problems really fast and get to almost the very, very close to the ideal solution.

Communicating messy data to leadership

Cool. Thank you for sharing. I see. Runwa, you had a question a little bit earlier, you want to jump in? I think it was around communicating messy data. To business stakeholders. I can ask and if you want to add more context, feel free. But it was, is there a better way to communicate to leadership or business stakeholders about most of the data being messy and just deal with it and build a model and get something from messy data? Usually 60 to 80% of data science time is data wrangling. However, leadership may want to cut this time. What do you think? Any thoughts to share?

Hmm. Man, I see both sides of this coin, to be honest with you. Right. And which I think is probably is at the root of what makes this a challenging problem. Right. You have what I talked about earlier, garbage in, garbage out. Right. Like the accuracy of your model, which I'll use as like a pseudonym for value. Right. Assuming more accuracy is more value. Right. The data going in that you have available, whether it's messy or even the scope of the data, like what you have is clean, but you're missing key pieces of information. That impacts your accuracy. And it's going to therefore limit the amount of value in some cases that you can deliver on.

On the other side of that coin is leadership has a point that says like, hey, we're not looking for the maximum amount of accuracy that we can see from your model. We believe that there's value in a lower accuracy model. And I think the answer to this question kind of lies in establishing a baseline and really understanding what you're trying to what you're trying to beat. In most cases, it's like a human decision. Right. Or a very basic, like more like algebra type problem. Like, for example, our delivery times. Right. One simple way to look at our delivery times. If we didn't have if we didn't know how many drivers were available, how many people were working in our store, we didn't know information about the order or all the orders ahead of it on the make line. If we didn't have all of that, we could still the baseline model is just a look at the stores last 20 minutes. What was their average delivery time and just use that.

Right. So knowing that that's the baseline kind of helps you have that target. And then you can say, hey, we either do or don't have the data that we need to perform better than whatever that baseline is. I think that's maybe something that is more tangible to leadership. Hey, what we want to do is perform better than this. And, you know, we are not able to create a minimum viable model that that overcomes that based on the data that we have.

ROI, metrics, and external data

Thank you, Jared. I see you have two questions that maybe kind of go together. Do you want to jump in here and ask those? Yeah, sorry. Well, the first one is kind of like ROI is always a big conversation when we're talking about any projects, especially when you're trying to, you know, at the strategic level, try to which ones had the priority. And I think the other one was is more from a kind of metrics perspective of the data is do you what amount of external data thinking like whether any kind of stuff you would use outside of the company is the mix between internal versus external data were much of the questions.

Got it. Yeah, I think one of the skills that I had to develop as I transitioned from individual contributor data scientist to senior data scientist to people leader in the data science space is being really, really good at capturing the value chain. Right. Because in data science, you might not be doing the thing that then immediately impacts the business value. Like you might be doing something two rungs down. And I'll just like bring up the delivery time estimate as an example.

Right. Like, why does giving a customer a better delivery time estimate? Why is that valuable to Domino's?

Right. Well, if they get a better estimate, like we're more likely to meet their expectations. If we meet their expectations, they're more likely to order more pizza. If they order more pizza, then the company makes more money. That's like a really stupid, simple example. But like that can get really complex depending on what your business is. So being really good at illustrating and capturing that value chain, I think is an important skill to develop. And it's kind of an art and a science.

Internal vs. external data

The other question was on mix of internal and external data. I feel very fortunate at Domino's that we have a really, you know, what I think of as a really well curated data science mart, where we have access to a lot of information about a lot of information about our stores, our orders, our customers. And it's really in not perfect, but it's in a very good place in terms of like accessibility to our data science team. So the amount that we have to look at third party data is not huge. So it's definitely primarily we're looking at Domino's own data. But there are certainly areas where you look for third party data. You know, on the store siting team, I purchased geospatial demographic data.

So understanding where, you know, what are not our customers look like and where are they located and information about them. We also purchased like credit card data, weather data, event based data. So I would say predominantly Domino's, but certainly we bring in third party in certain cases.

Data science applied to recipe development

Joseph had a question earlier that was, you mentioned a lot of different areas of the business that the team touches, but has anybody applied data science techniques to recipe development or any A-B testing there?

No, that's really cool, honestly. So certainly our research science team is involved in, you know, consumer research around new products and yeah, anything that we, if you notice like Domino's doesn't take like a limited time offer approach where we're like, you know, putting out things and testing them like that. We really put a lot of effort into our product research and like really ensuring if we're going to release a new product that it's going to be a creative to the business and it's going to be well received by customers. So we're not like, we're not the brand to be putting out new products all the time, but when we do, they usually do really well because we do a lot of research ahead of time.

Now, this is like a really cool, fun, like LLM case study, I think is like, hey, if we looked at generative AI, like what kind of crazy pizzas or products could we generate from AI? That sounds like a fun one that kind of sparked my thinking on that. I see a lot of people volunteering to test pizza. You're in the Ann Arbor area. We do have product testing panels that are open to the public. So shoot me a message. I can get you added to the list.

Partnering with stores and operations culture

Yeah, absolutely. Thank you. First of all, thanks for being here, too. And thank you for introducing the pizza tracker, because outside of people comparing their product to Uber, that's the second that I hear is, can we build a Domino's pizza tracker for our data pipeline or something like that? So I used to work in telecom, and so we did a little bit of supply chain as well. We did like field measurement. And so I was wondering how closely your team partners with the folks that are in the field, whether it be the stores, the drivers, whoever it might be, to observe some of the creation of your data, because that's something that we found very insightful. Sometimes we saw things, like you talked about delivery duration or time, and we found lots of different ways that people could kind of, I don't want to say game the system, but change the creation of the data in a way that we weren't expecting.

Yeah, this is like one of the pieces of culture that really excites me about Domino's. Everybody that works for Domino's corporate offices knows how to make a pizza. So we have a program called Pizza Prep School. So when you're onboarded into the company, you participate, and we literally have a fully functioning Domino's store in the center of our world headquarters. It has glass all around it. You can see it from anywhere in the building. And when you're onboarded, you get with your onboarding group, and you run that store for a week. So, and that's everything. You take orders, you do delivery, because people in the corporate office can order pizza to their desk, you know, to take home with them or to have for lunch. You make, you know, you spread the dough, you sauce, you cheese. There's even a bubble popper position. So watching the oven and popping the bubbles as the pizza goes to the oven, and washing dishes. Like you do everything in that week.

And yeah, everyone from our CEO down, you know, to the lowest level team member knows how to make a pizza. Luckily, the CEO is better at running a company than making pizza. But yeah, that's something that's really infused into the culture of Domino's is that operations mindset and like boots on the ground.

Luckily, the CEO is better at running a company than making pizza.

Absolutely. Rachel, if we ever do a data science hangout in person, I think we've found our location. I know, absolutely. I think that's how this started, Zach. Weren't you offering to do a hangout in person? I'm, the offer stands. Let's figure it out.

Key business metrics

I see Ross asked a question a bit earlier. I'll read it. It was, what is the most important measure in the business? And what is most predictive of that measure?

Oh, my God. This is a hard one. Well, it's funny because it shouldn't be hard. So I will, depending on who you ask in the company, you might get a slightly different answer. Both of them are related, but slightly different. So one group would tell you that it's pizza sales. That one's pretty obvious, right? Sell more pizza, make more money. More money is good. Investors like that, funny enough.

But the other group, because this is FYI, like Domino's is a franchise model, right? So of that 6,800 stores that I mentioned, 99% of them are franchise owned. So I think we own like 200 or 300 corporate stores out of that 6,800. So some people might say, hey, our customer is actually the franchisee and our key metric for success is franchisee profitability. More profitable franchisees, more stores get open, more royalties come through, the more money that we make. So to get to your point on what are the correlates, right? This is where herein lies some of the gray area, right? Because if we're trying to maximize franchisee profitability, of course, pizza sales are related to that. But having more efficient operations, right? And cost savings also kind of play into that equation. So when you look at it from that lens, the operational efficiency and reducing the cost for franchisees to open a new store and to operate a new store become really critical. So that's my long-winded answer to your should be straightforward question.

Oh, that's great. I see Rick actually had a follow-up on how it works with the franchisees. And it was, are the franchisees supported by your team? And if so, do they receive the same level of service as a company-owned store?

This is getting into some nuance of the business, but the answer is yes. So franchisees pay royalties on the sales of their pizza. That kind of goes back to the corporate general coffers. But they also pay a technology fee. So data science also kind of operates in that capacity as well. Kind of servicing them through providing them with better technology. So that's kind of the connection in the charter that we have with our franchisees.

Languages and tools for MLOps

I see Sally asked a question, but she had to drop for another meeting. But the question was, what languages are your teams using for MLOps work? And what tools are you using to help with those workflows?

Yes. So I want to say I am not a programming language Nazi. So I think we try to be open and flexible to the languages that team members want to use. That being said, we've kind of landed on Python as our production software language. And really, the decision was made early on to kind of standardize on one language for our production level work. Just from an efficiency standpoint, getting the right, creating that reviewer, having reviewers reviewing one language, not multiple languages, and not having to write two sets of documentation for everything that we're doing. It was a really practical decision to kind of land on one language for the production stuff. But in terms of like R&D experimentation, data wrangling, we are customers of Posit Connect. So even like some web app type development, we do an R.

Favorite use case: Points for Pie

I mean, we all love hearing these pizza use cases. Could you maybe share one of your favorite use cases of the team? Oh, yeah. This one's pretty easy. And unfortunately, like it's an older use case. So like, but still my favorite. It was the Points for Pie program that launched in 2019. And if you guys didn't see it or don't remember what it was, was we would give you Domino's loyalty points for any pizza that you ordered. Didn't have to be Domino's pizza. And how you redeemed those points was you would go into our app and you would scan your pizza. So we built a pizza, not pizza model to detect whether that pizza was present and then you would get the points for loyalty points for that pizza. So that's one of my favorites. Got really good PR. Not everything that we do as data scientists gets to end up in a marketing window that gets seen by everybody. So that was a really cool opportunity.

That's funny. How did that idea come to life? Or where did the idea first come from? I don't know if I got it directly from Silicon Valley. They built like the hot dog, not hot dog model or something like that. I don't know if that's how it was sparked, but I like to think that it was.

Model drift and monitoring in production

Donald, I see you had a question. Want to go next? Can you hear me? Yep. Okay, great. Yeah. Thank you, Rachel. Yeah. So a lot of times when I put models into production, I have to get a lot of buy-in in order to do any type of extra validation afterwards, like relabeling a random set of cases every quarter to make sure there's no drift assessment. Have you had any difficulties doing any type of drift assessment or any type of reanalyzing of the performance after production?

This is a big area of opportunity for our team. I think we do do kind of model accuracy assessment and we get daily. The whoever data science team kind of owns that production model is kind of responsible for monitoring that model accuracy and flagging any drift that they see in our ability to make predictions. And we do have some cool use cases of detecting anomalies. So for example, like with our delivery time estimation model, this just happened recently. There's a new manager in one of the stores and he switched everyone's position code to driver. So I don't know if it was easy for him or whatever to do that, but we started giving really, really fast delivery time estimates for that store and they're like wildly inaccurate.

The customers were getting pizzas in like an hour or something and we would say that they were going to be there in five minutes. Crazy. So we shut down the model for that store. We turned the flag off for that store. So there's stories like that.

But where I want to take us from the leadership perspective on this group is really creating that single pane of glass where we can have like, I almost think of like the command center Star Trek kind of thing that is like, hey, this is my control panel for all my machine learning models that are in production and kind of being able to assess just the general health of the service. Like, is it up and running? Is the throughput what we expected? But also like, is the data coming in what we expected? And is the accuracy what we expected?

So we can like land there and say, you know, hey, Model X, Y, Z, is it green health? Model Y is yellow. We need to look into that. And then Model Z is red. We need to shut that down right now.

So we can like land there and say, you know, hey, Model X, Y, Z, is it green health? Model Y is yellow. We need to look into that. And then Model Z is red. We need to shut that down right now.

So that's my dream, my vision. Making that a reality is on the road map.

It looks like we successfully sold one Domino's pizza right now. Nessa is ordering a pizza right now, just so you know. Get the pan, okay? Pan pizza, that's where it's at. Hand tossed is great, but you know, treat yourself. You deserve it. Get the pan.

Model transparency with Posit Connect

Going back to, I think it was Sally's question. Bill had a follow-up about how results get presented. Do customers ever have access to Shiny apps? I guess I'll add or dash apps because I know using Python too, or Shiny for Python.

Yeah, one of the really cool use cases around Shiny apps or like our Posit Connect platform is really in the area of like model transparency and like demystifying the black box of models a little bit. So one of our use cases is, you know, when we're working on a new model, our business partners have a lot of questions around, well, how would the model react if X? Or how would the model react if Y?

Like how would the model react if we get a school lunch order and there's a hundred orders placed at one time? How would the model react if, you know, we had a snowstorm? Like what kind of labor would it recommend? Those types of questions.

So what we've done is for some of our forecasting models, creating a Posit Connect application that just has that model on the backend and user inputs for the inputs for the model. So the user can go and experiment and do these what-if scenarios and see what the output would be. And that's really helped us achieve some comfortability with our partners. And I think it's a cool use case of that platform.

Advice for PhD students entering industry

Azuka, and apologies for mispronouncing your name. I see you had a great question around advice for incoming or aspiring PhD students. But what advice do you have for incoming or aspiring PhD students in statistics that you wish you knew before you started your program?

There's two things I wish I could tell myself in my first year, you know, out of my PhD and into the industry.

And one is, and I think it's related to another question we had today, but it's related to kind of this model accuracy, right? When you're in academia, like your objective is to find the best fitting model and to really look in order to move the field forward. That's where your focus needs to lie. Really, it is building that best model possible.

But when you get into industry, the best model isn't necessarily the one that is going to be the most valuable, right? Maybe there's a simpler model that is more parsimonious. It's quicker to get out into production and start generating value. And there is a case to go with that simpler, quicker to deploy model than taking the time to do the full experimentation and really achieve the most optimal accuracy that's possible.

So I had a few like scenarios where I was just taking too long to deliver because I was in this academic mindset of I needed to find the best solution. So, and not the most valuable solution. So that's one area.

And then the other area is, you know, at least in my academic experience, like intellectual debate was really kind of something that you become second nature, right? You know, I really cherish the times in my PhD education where I got to, you know, challenge my classmates thinking and push back. And, you know, what came out of that is usually mutually beneficial to everyone involved.

You can't do that in industry. Not everybody is really thinking that way. And really not everyone is expecting to join a meeting and be part of an academic debate. So, you know, that is one thing I learned early on is like, hey, why is everybody like, why does no one want to talk to me? And you can't challenge everyone's ideas all the time. So that's another lesson I wish I knew.

Marketing campaigns and data science

Arati, I see you had a question on marketing campaigns. Want to jump in? Thank you so much for this today, Zach. This has been so helpful and interesting. I'm a marketing analyst, not in retail, but I was just curious, do you know what goes on behind your marketing campaigns? Like how they're targeted? Is it based on locations of customers and whatnot?

My kind of overlap with marketing is really only on the MLOps side, right? My substantive area of my team is more on the operations lens of things. But I work closely from an MLOps perspective with my counterparts in marketing.

The marketing department is really exciting. And the data scientists that work over there are great. Just like what a creative group. And the process of coming up with a new marketing campaign, the leaders over on the marketing side, they struck that balance of coming up with the creative ideas, but also partnering with our data science team to make sure that those ideas are viable and doing everything that we can before we release a campaign to understand what the impact is going to be.

And fortunately, it's like a lot of things is on the upside. I don't know if you guys saw the emergency pizza campaign that we just launched, which said, hey, if you order a pizza, we'll give you a free emergency pizza whenever you need it. So part of the analysis was, how many people do we think are going to redeem this?

Because we need to be prepared from a supply chain perspective and from an operations perspective, communicate to our franchisees that they need to have more people on the clock or order more dough because we're going to get so many emergency pizza orders. So it's a lot of that type of work. And I really think it's a bright star on our team is the partnership between data science and marketing. I think it's an industry leading partnership.

Engaging people on the emotional plane

Curtis on our team is encouraging me to push back on the academic style debate one to say, well, how do we engage in good discussion if that academic style debate is sometimes less welcome?

OK, ma'am. My PhD is in psychology. So you're going to get that. Sorry, but you asked the wrong person that question maybe or the right person. I don't know.

So I think what's really important is to engage people on the emotional plane first. People have an emotional attachment to their work. It's what they spend. Most people, I would say, see the people that they work with more than they see their significant other sometimes. And the work that you do is such a huge part of your identity. And the projects that you work on at work are become a part of your identity.

So when you're challenging someone's ideas, sometimes you're challenging their identity and the value that they view themselves. So really understanding that and meeting people on the emotional plane before you dive into the logical plane, I think, is a good strategy that I've tried to utilize. I'm not always perfect at it, but that's my pro tip.

So really understanding that and meeting people on the emotional plane before you dive into the logical plane, I think, is a good strategy that I've tried to utilize.

Career advice

So I guess that's inspiring me to ask the career question then because I'd love to get some more advice from you like that. Is there a piece of career advice that you've given to others on your team or that you've personally received that you'd like to share with us?

One of the best pieces of advice that I ever got from my mentor was you can't have everything all at once, right? You have yourself, your work, and your family. And when you're looking at your career and you're planning your career development, you need to look at it through the lens of your entire life and what it means in those other aspects of your life. Not just what it means for you at work.

Maybe there's a time when, let's say, you have young kids and you need to pull back at work a little bit and put more of your energy into the family bucket. And it might not be like that forever. You might then decide after a certain amount of time, you're going to put more of yourself into your work bucket to advance your career. And looking at your career development, through the lens of your entire life, and how it impacts not just you, but the people around you, I think is the compass that I try to use when I'm thinking about my career.

Unexpected challenges: leading people

I'm trying to scroll through and see if I missed any questions. I hate missing people's questions. So if you want to raise your hand here or feel free to re-copy and paste it into the chat so I can see it.

But I think one more I see that I hadn't asked was, Azuka asked, is there anything unexpected in your current role that you haven't seen before or have experienced doing in your training? And how did you use your skill set to navigate that issue?

I think the biggest challenge that I see, I know I've experienced this, and I see it in others as well, is if you're coming out of a technical background for your education, you don't get much exposure on how to lead people. And so I think that's a big challenge.

And when you go from a technical role to a people leader role in the technical space, that's often a gap that I see is like, how do you motivate people? How do you have difficult conversations? How do you align people's skill set with their interests? Or is that even important? Those are the things that I often see that are missing. The technical stuff is usually not the challenge is what I've found.

IoT and micro-optimizations

Well, I know we're getting to the end of time here, but I do see your hand here, Rick. So I want to make sure we ask your question. Do you want to jump in? Well, thank you, Rachel. And Zach, you're an excellent example of a data science professional and a great representative of your company. Thank you for sharing your remarks.

Quick question, with your focus, maybe it's your sub-teams, I'm a little unclear about that, but your focus on optimization and operations, do you make use of any IoT, Internet of Things devices for micro-optimizations? I'm aware in other settings where IoT devices have led to DS teams to discover tiny optimization opportunities, but then a million times a tiny optimization becomes a big optimization. Zach, your thoughts or experiences on this?

So we do have data points at that level, right? There are items in the store that employees kind of engage with that then end up as data points in our database. For example, when they clear the make line, like they made a pizza, before they put it in the oven, they have to click a button that says I'm done making this pizza.

And that's like a really important data point for us because it helps us optimize points in the store.

So I think to expand on this is like, of course we want more of that. Never ask a data scientist that they want more data because the answer is always yes. So I think we would love more data from IoT type devices, but we face two challenges there. One is cost, right? It's not like IoT in a factory where you might have like 200 different steps that you can kind of throw a device on and get a good pulse. We need to do that across 6,800 stores that are geographically dispersed literally everywhere in the country. So like getting those IoT devices in the store is a logistical challenge and a cost challenge, right? Because those devices are usually not cheap.

Never ask a data scientist that they want more data because the answer is always yes.

Secondly, like our stores are franchise owned. So we would not only need to overcome those logistics and costs, but we need to convince franchisees that there's benefit in them doing that.

Well, thank you so much, Zach, for joining us today. I've loved this discussion and thank you all for the great questions as well.