Resources
Jan 8, 2024 |
1h 2min
|
1.3k views

Estimating & communicating ROI for data science projects

A recurring topic in Data Science Hangouts, meetups, and 1:1 chats is the ability to estimate, measure, and communicate the Return on Investment (ROI) for data science projects to business stakeholders. A specific question was: “Are you ever asked to estimate the ROI from an analytics project before the business is willing to invest in anything?"

I asked the community if anyone had something to share on this topic, and they delivered! A special thank you to Derek Beaton, Director of Advanced Analytics at St. Michael’s Hospital and Joe Powers, Principal Data Scientist at Intuit who kicked off the group conversation last month by sharing their own experiences.

Tips for communicating the ROI for data science projects:

✨ Engage stakeholders & identify gatekeepers
✨ Align evaluation metrics to stakeholder objectives
✨ Establish clear success criteria
✨ Anticipate where things can go awry
✨ Consistently communicate your central message
✨ Prioritize what’s most impactful
✨ Promote transparency while measuring project performance
✨ Ensure the benefit is unquestionably worth the time investment

Additional ROI Resources Mentioned:
✨ Analytics Power Hour Podcast - Estimating the Effort for Analytics Projects: https://analyticshour.io/2023/10/31/231-estimating-the-effort-for-analytics-projects/
✨ CDO Matters Podcast with Malcom Hawker: https://open.spotify.com/show/2OJXB8v32CrKbbsv9Uq2vn
✨ How to Measure Anything by Douglas Hubbard: https://hubbardresearch.com/7-simple-principles-for-measuring-anything/

If you want to continue conversations like this, we invite you to also join us at the Data Science Hangout every Thursday from 12-1 ET. Every week we're joined by a different data leader from the community to share their perspectives and experience.

You can add it to your calendar using this link: pos.it/dsh

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hey, everybody. Welcome to our hallway chat. I haven't come up with a final name for this, but this is our experiment to see if y'all like these. But my thought process here is, you know, how some people say their favorite part of conferences are those conversations they get to have in the hallway, or when you get to talk with someone about something you just learned or how it applied to you, or maybe something you wish a presentation covered.

So in one of our data science hangouts in early November, somebody had asked, Are you ever asked to estimate the ROI return on investment from an analytics project before the business is willing to invest in anything? And a shout out to Nick, who asked that. I'm not sure if Nick's actually here on the call now. But how do you approach that? And so there was a lot of conversation about this in the chat and people wanting to talk more about it.

So here we are. I posted on LinkedIn to see who'd be open to sharing their process for this and would love to introduce two leaders joining us today. They'll both share their thoughts to kick off the conversation. But this is a group discussion about all things ROI, to share ideas and our own experiences to help each other. Like the data science hangouts, it's totally okay to just listen in here. But if you have follow up questions on the topic or want to provide your own perspective on what's worked or hasn't worked, there's a few ways you can contribute.

So you can raise your hand here on Zoom. And just to tell people how if you're not familiar with that in the bar below, there's a reactions button. And from there, you can raise your hand. You can put questions in the Zoom chat. And feel free to just put a star next to it if it's something you want me to read out loud instead. Maybe you're in a coffee shop or something. And then lastly, we have a Slido link, which I will share here in the chat in just a second. But you can ask questions anonymously there as well.

And thank you to somebody who just said I need to switch my background because it needs to mirror the faucet logo. It's showing okay on my side. I put a virtual background because I'm at my parents' house and was hiding everything behind me. Did that last minute. But just to let everybody know, like the Data Science Hangouts, this session will be recorded. So if you want to go back and listen again, I'll be sure to share that to the faucet YouTube. But before jumping in, just a reminder, we're all dedicated to making our community events a welcoming environment for everybody. We love hearing from everybody, no matter your years of experience, the languages that you work in, titles or industry. But with all that, thank you so much for spending time with us today.

I'd love to introduce both Joe Powers, Principal Data Scientist at Intuit, and Derek Beaton, Director of Advanced Analytics at St. Michael's Hospital. Do you both want to maybe kick things off by just sharing a little bit with us about each of your roles and kind of say hello before we dive into ROI? Derek, do you want to lead off?

Sure, thanks. So I'm Derek, Director of Advanced Analytics in a unit at St. Michael's Hospital in Toronto. Our unit is Data Science and Advanced Analytics. We are an applied healthcare unit. So we put StatsML AI tools into production for mostly clinical use, but there are quite a few operational or like staffing solutions that we put into place. We're not research. It's all about getting things actually done and into the hands of users. Overall, our group is about 30 people. We technically work across three hospitals, St. Michael's, St. Joseph's, and Providence, all in Toronto as part of the Unity Health Network.

We are mostly in our house. We have maybe 70% of our solutions in R, and then we've got a variety of things in Python, including some new stuff we've recently launched with a little bit of stuff in pretty much any sort of analytics AI and ML domain. So imaging, clinical prediction, patient flow, forecasting, you name it, we probably do a little bit of it.

Awesome. Joe, you want to go next? Yeah, sure. So I'm Joe Powers. I work at Intuit. You're probably familiar with some of our flagship products like QuickBooks, TurboTax, MailChimp. And yeah, so I lead experimentation, which has a huge amount of ROI involved in it, largely around investigating new methods for how we could run faster and more accurate A-B tests for product and marketing decisions.

And so when you move a whole company, you're shifting their decision making process to a new statistical method, people want to know, why is the juice going to be worth the squeeze on this? And so, yeah, when Rachel brought this topic up, I was happy to join in because it's been, I feel like it's one of those hidden curricula topics that it's central to doing your job as a data scientist, but absolutely no one teaches this. So good for you for being here today.

it's one of those hidden curricula topics that it's central to doing your job as a data scientist, but absolutely no one teaches this.

Kicking off the ROI discussion

So I was thinking we kind of jump in and get started with sharing a little bit about how you both have approached this, and then people can jump in with questions and thoughts too. I know you both have very different approaches to this too, and it's really cool to see the difference across industries as well. I don't know who wants to get started, but I was thinking maybe kind of giving us a baseline and sharing a little bit about what you do. Derek, do you want to maybe start with explaining how the process your teams roll up?

Sure, yeah. And in some ways, maybe this will feel really specific, but that's because we're in healthcare, in a hospital, but I think a lot of this is pretty generalizable. So, and I'll share a document or part of a document with us in a moment.

So when a project comes our way, it usually comes from someone on the ground. So a physician, or someone in HR, or say a clinical manager in some way. They have a problem, they know data can solve it, and they work with a few of us in the team, primarily the directors and the product managers on filling out and detailing a project intake form. And one of the most critical parts of the project intake form, which I will, Rachel, am I able to share?

Yes, I will just make you a co-host so that you're able to. And I got rid of my Posit logo, which was backwards in the background. I'm on the boardroom here for this ROI conversation. Awesome. One second, I'll make you co-host. There you go.

Awesome. Thank you. So I'll just do this real fast, and you can take away my co-host abilities in a moment. You can keep them. I just didn't want everybody to be able to screen share.

So as part of that project intake, there's a key section here called evaluation metrics. And right off the bat, you should have a good idea of whether you're trying to address mortality, admission, length of stay, financial issues, or human effort generally across the types of problems you want to solve in the hospital. There's not a box other, which is usually an indicator of like, we probably aren't going to take this on or it's hard to define or hard to measure. But a lot of these are generally easier to track or measure and assess the impact.

And from there, we work with our collaborators on how to define the primary metric. So are you trying to do something to reduce the length of stay? Then we are going to look at how to do that as well as tracking the actual length of stay pre and post deployment. And when we do this, we have discussions early on about what the expected level of change is. So if you think you can bring length of stay down for, say, all patients in a certain unit by, let's say, 30 minutes in like an ICU or something. So you bring the overall down quite a bit. We'll get this written down here. And that's how we proceed and measure and assess things when it's after the fact.

I'll stop sharing now. But we have fairly deep and detailed discussions at this intake stage about what the metrics are, how we're going to measure it, how we're going to track it, and what the level of change is expected to be. And a lot of the expectation will come from knowing the hospital or the system in general, knowing the types of changes that do exist in other sites where you can point to literature or something else somewhere else and say, look, they saw a 10% reduction. So we would expect about the same. So a lot of this is based off of targeting something, identify a key healthcare economics metric, and let's work out how much we think we're going to actually make a difference.

If things aren't, say, a huge benefit or if they fall kind of in that other box, this is where we say, like, maybe we're not going to take this on or we have to dig really deep into the types of things that might yield value overall for either the unit, the hospital, the system, like the whole network of ours before considering taking it on. So we get pretty deep into this on how we actually assess and decide what to do before we fully commit.

As part of this where we won't necessarily, like, Rachel, so I guess to one of your points, like, before any commitment, there is some commitment we have or that we'll get into. So if people come to us and they're like, yeah, I think we can get this measure here, here, or here, and we want to get a baseline, we'll absolutely work with anyone to establish what those baselines are. So that part comes with the project intake. It's just the deciding to move into the full development of something is really where we need to see what that baseline is and really dig into what these improvements are going to be.

I'm sorry if you said this already, but how long have you had that process in place with the intake form? Quite a while. The intake form here has been revised as recently as a little over a year ago, but those metrics have been in place pretty much since the start of this group, which predates me. So always trying to target big healthcare economics type metrics.

Well, I was just asking in the chat, I was curious for people listening in if you have something similar to that, like a project intake form, and if so, what some of those evaluation metrics might be for you. As maybe a good reminder, I should have said up front, but I love to see everybody talking in the chat during these sessions too. If you've been to a data science hangout before, you might see a lot of resources get shared in the chat. So feel free to use that during the session too. Joe, do you want to share a little bit?

I can just kind of echo some key things that Derek said around identifying those key metrics. A lot of people talk about stakeholders. I tend to think about gatekeepers. You have lots of stakeholders, but really thinking now about which ones are the gatekeepers that are going to block or allow your project to roll out? So what are the metrics that they really care about?

And yeah, establishing baseline. There's very likely a dashboard that those leaders are regularly referring to that you can anchor to as a shared reality. And then what is a reasonable lift that you could expect? That could be based on literature review. It could be based on qualitative testing. If you looked at 20 users in a UX setting and you said, oh, our invoice completion rate went from 20% to 90% with the new invoice flow, that is actually a great anchor point that you could now start thinking about. Like, okay, that's in an idealized setting, but when it moves into the product, what's reasonable to expect?

So a lot of times you're just dealing with the best data or best information you can get your hands on. And that can run the gamut from maybe it was captured in your data lake to running a few surveys or doing some qualitative sessions. But any information is better than none.

Data simulation for ROI estimation

So yeah, key metrics, baselines, reasonable lift. And then actually that's where for me, it becomes a data simulation task after that. And I'm happy to share a little bit because nobody ever taught me how to do this, but it's just become my go-to for the workflow.

So I'm in RStudio right now and I've loaded the tidyverse. And so you can imagine, yeah, my baseline is this rate A. So maybe I have a 50% rate. Maybe this is a conversion rate or a click rate, whatever is that key metric for your gatekeepers. And maybe I'm just going to explore some reasonable lift. Maybe I think I could lift this a 10th of a percent with some new program. I've got 100,000 customers and I'm going to run 4,000 simulations.

So yeah, I can execute this now in this cell. So I'm using the binomial distribution. If you're not familiar or you're not comfortable using these likelihood functions, I think this is something just really worth leaning into. A lot of abstract statistics become really concrete and intuitive once you get comfortable simulating data with likelihood functions. So all I'm doing here is I'm running 4,000 simulations with 100,000 trials, and I can set the probability of success on each trial. And this is going to output a tally of events. But then once I divide that by the population,

I'll always get a rate. So maybe this outputs 50,000 events, but when that divides by 100,000, now I get a 50% rate. So I could actually just give you a peek at the distance.

So yeah, for my first simulation, I have 50.1%. The second one, 50.2. I hope this is getting clear. I tried to keep this minimal. Now would you be able to describe what's a project that you might be working on, where you would be doing this? So this is really very close to the idea of a power analysis. If you were introducing this 0.1% improvement, you could start asking questions how likely are we to detect it?

And so even in this case, if I just look at how often are my B simulations bigger than my A simulations, that's not saying it's statistically significant. It's just saying bigger at all. The smaller the change in 100,000 users, 68% of the time it would be larger. And you could explore the range of differences that you might observe, like a 95% interval of those potential differences. The very concrete examples that I could give are like when I'm exploring a new A-B testing method, what I'm going to do is I'm going to simulate the test data from 100,000 A-B tests. And then I can start asking very practical questions from those simulated data, like how often did I detect the superior treatment condition? How long did it take to detect the superior treatment condition? How many additional conversions did we yield because we detected the better experience earlier and rolled it out to 100% earlier in the season?

Those are the kind of return on investment questions that I can explore using very basic data simulation that my gatekeepers, my stakeholders really care about. They want to know before they green light using a new statistical method in season. They're going to want to know, does this detect benefit more often and faster? Does it detect harm more often and faster? And that's just all. This is all just that initial return on investment research. They're going to want to know the same thing on the tail end of the project about how well this lived up to your projections.

But I just wanted to show this as using these likelihood functions that you have available in NumPy, you have available in BASAR. But they're very powerful and very persuasive when you're trying to do ROI sizing for a new project. That's great. Thank you. And thinking about that, like the ROI that you showed there, how do you translate that into communication with the business stakeholders? Is it through a business case document?

Communicating ROI to stakeholders

Through all means possible. So it's, you know, again, as I mentioned gatekeepers earlier, like thinking about who's approval, who's buy-in you need for a project to get green lit. That person is going to have a lot of people demanding their attention. They've got a lot of fires to put out. Like, how are you going to get your idea in front of them so that it sticks?

You know, you give a 20-minute presentation, you know, 20 minutes later, they're probably if you succeeded, they're only going to remember one or two things from that presentation. And so, you know, the like clarity of your communication, and we had a former CEO who loved the expression, repetition doesn't sully the prayer. You know, just repeat, repeat, repeat, and just keep kind of driving home that central message that appeals to that. Like if they have an urgent problem and you're offering the solution, if you're doing it clearly, you're not going to have to offer it that many times.

repetition doesn't sully the prayer. You know, just repeat, repeat, repeat, and just keep kind of driving home that central message that appeals to that.

But be ready to deliver. And so anyhow, but yeah, through communication, it's like, you know, start sending word out in Slack channel, offer, you know, I guess more peer supportive trainings through, you know, Zoom meetings and live meetings, get your peers on board, you know, try to get, work through your manager to get, you know, FaceTime in, you know, there's usually recurring executive presentations where you can like get in front of your leaders to pitch, you know, your potential solutions. And if you're coming in there, not only with, you know, the performance of your solution, but the return on investment in a metric, they care about and recognize, like, this is just all stuff that like greases the skids to make sure that, you know, your project gets a mandate and goes into practice.

And like we said earlier, just be ready on the tail end to verify that you delivered what you promised because that's, that really kind of like puts a bow on it and, you know, and really like cements the relationship that, you know, in the future, it's just going to be that much easier for you to get that FaceTime, to get the approval to put your projects into production. Definitely. I see, George, you asked a question in the chat a bit earlier. Do you want to jump in here?

Estimating time investment

Okay. Yeah. Sorry. I'm battling a cold and my voice is pretty terrible. Yeah. I was wondering how you would estimate or measure your own time investment, because I think the other half of this equation is, is understanding how much effort it will take for you to, to do that. And also like how accurate those estimates are.

I have only seen that accurately, accurately measured with survey data, like calendar day. I've, I've tried to use calendar data for these kinds of projects. It is meaningless. Like, you know, the loyalty of people to their calendars, the accuracy, the coding, just don't even try it would be my advice. But just, you know, asking like a good survey question about like, you know in the past week, like asking, you know, surveying analysts, for instance, or surveying the key people, like in the past week, how much time did you spend on like these parts of your work? Is that a typical week? But usually it helps to like anchor, like anchoring surveys to a specific time period can really improve the accuracy of responses. But, but, but honestly, just survey data is the only thing for, for your, to answer your question that I think that's the best signal. It's not great, but it's, I think it's the best you can do.

Derek, do you want to add anything there? Yeah. So estimating ahead of time, we don't like, that's just, it's really hard to make a guess, especially when a lot of stuff we do is pretty variable and unknown, especially in the early like parts of the projects. Measuring later, we're probably a little bit better about once we're like really in the weeds. And this would come a lot from our product managers and the way we've set up like our sprint cycles. Got a pretty good track of, or at least within good, like small windows, we have a pretty good sense of what we're doing and how much effort we're putting into the respective projects.

But this is definitely not something we, I would say estimate very far in advance. So like at the, the ROI project intake stage, more so like in the project, is this going to take you a week, two weeks, one day, stuff like that. But I think where maybe that question, if I would make a guess, George, like is this about like trade-off? Like how many person hours is it really going to take? Is it going to take like three, four years for something to like happen? And maybe that cancels out some of the ROI from the project.

Yeah. Yeah, exactly. If it's going to take you, let's say eight months, you know, to make this 10th of a percent increase, is that worth it? I mean, and then other factors, you know, like, well, maybe in a couple of years, we know things will change. So maybe that 10th of a percent isn't even going to last us that long because things will change and then alter that. So. Yeah, I didn't use 10th of a percent because it's like a good target. I use 10th of a percent just because I was writing this five minutes ago. But you should but you should have, obviously, I think a lot of times the benefit needs to just be unquestionably worth the time investment. You know, like it's, you know, it often just doesn't come up because it's just so obviously like the, you know, the reward to investment ratio is should be high enough that it's just not a hang up. If it's if it is getting that marginally close, like it's probably not something to pursue because let's face it, most ideas fall flat.

Competing priorities and Bayesian thinking

Yeah, I guess I'm kind of like really interested in this question because we have a small data science team and we get a lot of different we have a lot of needs. We have a lot of requests and we have we support the whole at JPL where you got a lot of a lot of different needs, but we also just have very focused areas. So we kind of have to like run things through like, well, does this support the vision, you know, like our vision statement here? Does it support the strategy? But also, yeah, we have a 10 10 projects that are great ideas, but which of those 10 would be the best ROI would have the best ROI. So it's it's all about like, I think, for me, competing priorities. So with a limited amount of resources,

I have found it really helpful, I guess I've gotten more into Bayesian statistics. You know, this idea of a prior probability, particularly around success. And, you know, I think this is I miss this to me maps exactly with your question. It's like, you know, you have multiple potential projects you can pursue. And, you know, it's like, you're probably like, you know, you and your stakeholders are like, really the experts in this space of the company, like, what a prior probability of success around it, like, you know, what is the range of outcomes that you know, going into the project, like, you know, it might realize, you know, it might manifest as and then I think that could be enormously helpful in helping you decide, is this something we should like explore more? Or is this something that's like promising enough that like, it's time to like, go do it.

But again, like, you know, I think, prior to my, like, getting into Bayesian statistics, that idea would not have been as clear to me that like, oh, I could actually like, go into a project. And I always do, in fact, go into projects with a prior estimate of like, how successful it's likely to be what the range of effect sizes I might reasonably expect are, I just wasn't making it explicit before I learned Bayesian statistics. But it's hugely important. Like, you know, if you if everything you do has a low prior probability of success, you're just generating noise. So I appreciate that.

Compliance and regulatory uncertainty

I'd love to pull up with someone just about to ask a question. I was I can't figure out how to raise my hand today. Sorry about that.

Have you guys ever run into an issue where you have to make a change because maybe it's compliance or legal and you still have to then estimate like the amount of time it's going to take to do that where, like, the return on investment is not necessarily like there may be dollars saved to do this, but you also are legally required to to make those changes. And so then you have to kind of like plan out your project from that perspective. I know working with hospitals, there's a lot of compliance regulations. And so curious how you guys approach those kinds of those kinds of requests.

If we have to make a change within one of our own projects that we just we make the change. There's no no discussion really about how long is this going to take or what is the what is the return here because we've already got the return or know what the return is or has been from that project or that that initiative. So in the case of something where it's like compliance or security or critical upgrade, we're just going to do it like there's no question about it. We're just going to do it.

Yeah, I've run into issues where, you know, we're trying to do something because it's a potential compliance thing. And so you get into that hairy thing where it's like, OK, well, maybe it's fine, but maybe it's not fine. And if it's if they decide that it's not fine, then it's a problem. But that like living in that fuzzy area where we'd like to invest something earlier than like before, it's a huge problem, like becomes a weird thing because saying like, well, this could be an all or nothing situation if we are noncompliant is not really something that I've had a lot of success bringing up to kind of prioritizing those projects.

If it so I want to make sure I understand your question. So obviously anything that's like a question of legal compliance is just like, yeah, good to do it. Well, sometimes we get we get documentation from government partners or regulatory agencies that's like, hey, these are proposed changes. We don't know if we're going to lock these in yet, but it could happen. Like we could decide when we release the final version of this to do this. And so you have like a six month window where you've got to like size up what what that difference is.

OK, OK. So if I understand this better now, I'm sorry if my answer sounded pedantic before I was like, you know, I was hoping it would lead into something more. But. So if there's this potential thing that could happen, it's what you're saying. Yeah, that could impact you. OK, so again, I find this Bayesian idea of an expected loss really helpful. OK, and this is when you don't know, like the future is uncertain, you know, like you don't know what's going to happen, but there's some probability it's going to happen and there's a range of penalties or losses you could incur if it does happen. And like a real simple example is that like if there's a 10 percent chance you're going to lose a dollar, your expected loss is 10 percent of a dollar, you know, 10 pennies versus a 90 percent

chance you're going to lose a dollar. That's a 90 cent expected loss. And so you can I think like given that uncertainty, it's not a question of like breaking the law, but like your question is around like, oh, we don't know if there's going to be these regulations in the future. How much might this impact us? I think this expected loss framework can really help like support rational decision making in an uncertain future rather than always jumping to the most extreme worst case scenario, which is not an optimal way to make decisions.

I want to echo what somebody said in the chat, and it's a phone number, so I don't know who it is, but it does seem like we're discussing different, they said, and equally important methods of establishing ROI. And I'm realizing that as well, that I think in my head, I always thought of like ROI as like you have to come up with this like accounting number and like the actual number that is going to impact in terms of like revenue versus thinking of the like statistical side of things as well. So it's really interesting.

Handling underperforming projects

There was a good question in the chat, Rachel, about like how to handle when your project underperforms relative to projections. I was just going to have Jadie ask that one live. Jadie, if you want to jump in here.

Yeah, sorry if my treadmill is a little loud. I don't have my headphones in right now. No, we can't. We can't hear it.

And shout out to the cats who code t-shirt.

Yeah, I guess I was just wondering like how you handle if a project that you had to pitch and sort of fight for how you handled, maybe it under delivers or underperforms, as you said. Mostly, I guess I'm wondering about like the trust aspect of like the decision makers or stakeholders, how that impacts any future proposals you have for projects.

So you got to remember, A, like there's variability in the world and B, you can't control everything. And the thing you can control, though, is transparency. And so I think that's like just really key that like one, you know, your findings, so your ROI projections should reflect a range to begin with. You know, I mean, even just like mean and interval is like a great anchor point of like just to remind them like we don't know the future with certainty. Like this is the like reasonable range of outcomes we might expect.

And then like just to, you know, be transparent throughout about how you without drowning them in detail, but just be transparent about like how you arrived at your numbers and then like what you saw at end of season. I mean, there are certainly been projects, even successful projects that have been on that along the way fell short, you know, because we we were unaware of some key factor. You know, and then you learn because it doesn't deliver the way that you expected it to. And never during that did I feel like the leaders who had invested in us were upset or frustrated. It was like, look, we leaned into a difficult problem and we learned along the way. And these are, you know, to be clear about like this is what we overlooked. This is the correction we put in place as we pursue it next time. And I think that that's that transparency and honesty is like the most you can do.

And I think that that's that transparency and honesty is like the most you can do.

But again, just also anchor them to a just get them thinking ahead of time about like there's a range of outcomes we could experience, even if we did introduce this through improvement of, you know, 1% or 2%. Like it's sampling variation. It's it could show up in our, you know, customer sample for the year as a lot of different values. That's not how normal people think. Normal people think very like causally, narratively, precisely. They don't think in terms of variance. So you have to kind of condition your key stakeholders to that, to expect that kind of variance and they'll they'll appreciate it in my experience.

I want to remind everybody, if you do want to provide some perspective here too, and weigh in on what you're doing, feel free to raise your hand, just so I know to call on you. It's in the zoom bar below. If you click on reaction, we'll have a raise hand. Sorry, Derek.

No worries. It's great. So to the underdelivered part, we, this is probably gonna be very different to what Joe said too. But like we, we have maybe like four different ways in which we can expect and or watch actual potential or actual under delivered projects. So early in our baseline, like trying to establish what the effects really might be. If it looks like it's not quite as big of, say, an issue or feasible for us to pursue it, and we suspect it would be something that often doesn't happen, then it's not quite as big of a and we suspect it would be something that under delivers on the potential impact, we're not going to do it.

During model building, if we do not cross certain acceptable thresholds, in particular, things around like precision recall or PBV and MPV, like we have some deployments like do not miss, like you just absolutely do not miss a prediction. If the models aren't performing well enough, like we will not go forward with it, which seems probably pretty typical. But then we get into a little bit later where, say, in kind of like an earlier silent deployment, we can watch things for a while, be very cautious and safe for some of our stuff. But then, and I think Chloe's here, so she can definitely jump in and talk a lot about this for some of the work we do. And sorry if this kind of gets a little maybe MLOpsy, but we have two types of monitoring we do. One is we monitor models and we monitor the intervention. So the model may target one thing while its use is meant to impact something else. And those are the primary metrics that I showed.

If we're not seeing, say, an actual impact, or something is going in the wrong direction, like that for us is underdeliver and you just stop. So we are also carefully watching the intended intervention metrics here, which is the primary metric, so like what you're going to change in the hospital. And fortunately, we don't really have anything that underdelivers, because we do, like Joe said, we understand variability, like things are very weird, especially just a smidgen outside or still within the pandemic in healthcare, like things are all over the place. But we've been talking a lot about this in the context of one of our particular projects where in the post phase, it's been deployed, it's being used, made a big impact, dropped readmission rates, and we're seeing over time it's drifting back up. So we have this really uncomfortable question right now. If we pull it, does everything just jump back up, or is this a natural trend back up where something about the deployment a year later is not quite actually working? So yeah, we have to start thinking about this. We didn't really underdeliver, but we're starting to lose some ground. And we don't know if it's actually like what we did or if it's something bigger, which is probably the case within an exhausted healthcare system, where like readmission rates are just kind of up in general. So this is actually super tricky for us because we're at a point where we need to figure out as the direction of an intervention is kind of losing some ground, what do we do?

I'm thinking also, as we talk about this, that like, you know, we mentioned earlier, like those metrics that your stakeholders all like align on and value. I'm thinking also that like just explicitly redefining those metrics in some cases is often really valuable. And this goes to that like underperformance issue, is that two people can mean completely different things by the same metric. And if you both explicitly defined it, and then thought ahead about like, what are potential ways where, like, I like Derek's monitoring question or monitoring answer a lot, like, what are things, what are ways things can go awry? And how can you get ahead of that?

So like, you know, you think about A-B testing or experimentation and science, like, science really cares about false positives, like they don't want false positive effects getting into that like body of, you know, of scientific knowledge. I think that's a lot less important in industry. It matters in medicine. So I'm gonna, I'll say to Derek's case, like this is not his approach. But if you think about like, in tech, a lot of times a false positive is a free mistake. Like all the development costs are sunk by the time that the test runs. If you release, you know, some minor change that turns out to be like a false positive result, no one's harmed by this. It's actually the false negatives in tech that are really dangerous. And those are not the priority in statistical significance testing in other areas.

The reason I'm bringing all this up is like, so I work on experimentation, like things that I get metrics that matter to me and my stakeholders are like accuracy, and duration or speed. So like, well, what's accuracy is accuracy, avoiding false positives, they're free mistakes. Or can I redefine accuracy as like, some kind of optimization rate that's focused more on detecting true effects than avoiding false effects. Like that kind of shift accuracy is like a term we can all throw around. But like, I've, you can completely redefine that and justify justifiably, but you need to get your stakeholders on board that like, hey, look, we're not going to key your index on like false positive rates. You know, that's not what matters to us as a business. And again, if you've explicitly addressed that up front, and you say like, look, I think we could, we'd have better business performance by shifting our focus for true positives, rather than avoiding these false positives. And then again, you know, you do all your simulations to show why you've got a promising way to pursue this. But you know, you just admit up front, like, we're gonna have more false positives. And we're okay with that, because those are free mistakes in our area. Those are not free mistakes in medicine, but they're not free mistakes in a lot of other areas. But in many areas of tech, they are. So yeah, just kind of thinking in advance about like, what are ways that things could go awry or get misinterpreted, and then making sure that you're, you have the tracking system in place, which is also just all going to come key at the end of the project when you have to show like, how do things, you know, materialize in real life?

Tying data science back to strategy

Yeah, definitely. And Dan, I see you have something to add to this topic, too. You want to jump in? Yeah, although it might be changing the subject a little bit. So I don't want to change if that's not what you want to do.

Oh, that's okay. But does anybody else feel like this conversation could go on for three hours about? I feel like there's so much to talk about here.

Yeah, no. Sorry, I'm battling a cold, too. I feel like I'm the only non-data scientist on the phone. So the conversation about Bayesian statistics a little bit. Good. Thanks, Rachel. Make me feel good here. You know, the things that I'm thinking about are maybe a little bit more fundamental. I think what Derek and Joe are talking about are like, really high level. I mean, this is like, professional. I know, Joe, you said you've never been taught this, but I feel like you could teach the class on it. The things that I think about... How to learn something in just 10 years. Yeah, exactly. What I think about this is, you know, what does the company need?

You know, and I think Derek's example that he showed was just so perfect of this. Because it didn't do it on purpose, or it did on purpose, but it wasn't like in your face. But it was like, these are the metrics the company actually cares about. How is what you're doing fixing those metrics? So I think what we get a lot, what I tend to see, is passionate data scientists. And they're going off, and they see stuff, and they're like, oh, I bet I could do this. But it doesn't come from this direct line from strategy. So I love Derek's presentation of it, because it's tell me where it is here. So I think that is, if we had to take one thing away from it, I would love to take that away. Yeah, no, I 100% agree. Put it in the metrics that your non-technical stakeholders care about. Yeah. Which is probably not area under the curve. You know, it's like, live saved beds, empty beds, et cetera. Yeah. Put it in those terms. And the data scientists will be compelled to. So let's just go for that lowest common denominator for persuasion.

Put it in the metrics that your non-technical stakeholders care about. Which is probably not area under the curve. You know, it's like, live saved beds, empty beds, et cetera. Put it in those terms.

And then my next point, and then I'll get off my soapbox here. My next point was, why are they asking about ROI? And you know, sometimes they ask about ROI because it's a multi-billion dollar decision, right? And so the CFO needs to know, right? And so there, 100%, you need to go dig into this. But I think sometimes they ask about ROI, because they don't think it's valuable. And they're like, just asking you, prove your value on this. And then I think, you know, sometimes they're a project that are devoid of any kind of corporate metrics or anything like that. And so they're saying, what's the value? So you can do all the work you want. You're never going to convince them. Okay. So you got to pull it back to strategy somehow and bring that forward.

I guess I don't see those as competing. Like if you have a strategy, a good strategy should be anchored to like a clear goal. And then the company or your leaders are aligning resources to ensure the successful pursuit of that goal. I guess I don't see the conflict at all.

No, no, no. The conflict is with your, it sounded like your organizations are like very structured. And I think maybe some of the science-y organizations, it's a little bit looser. And so you get a little bit more exploration and that's not very easy to tie back directly to strategy. This is a good point. This is a good point. Some things are like, like I can think around like skill development among the data scientists and analysts. Like it's something where it's like intuitively, it's like, yeah, we invest in this, like this should pay dividends. That will be very, very hard to measure, but it's like, or, you know, the best measurement you're going to get is probably anecdotes and case studies, but is it still worth doing? Yes. But like, again, you're going to have to be transparent

upfront that it's like, we're not going to be able to measure this with like, you know, X percent increased conversions or decreased mortalities, but it's like, where do the moonshot ideas come from? You know, like again, it kind of depends on like, you have to know your leadership and like, what do they have appetite for? Like if you have, if they have some like visionary bandwidth where they're willing to float some of those like moonshot, they're like education's a moonshot idea, but you know, like if they're willing to entertain some of those less clear investments, good. I think that could be, you could frame it as part of a good investment portfolio. But yeah, you can't measure those things nearly as easily.

My point was just to say that the people trying to do those moonshots might get a lot of questions about ROI, might do as much work as they can and still never make it. So when you get those questions, I think it's just worth thinking, why are they asking me this? You know, how do we position this? Is it possible to convince them?

Something I thought would, might be helpful to add too, is sometimes they're not asking for the ROI. Like you might be sharing what you think the potential ROI is to help push your own idea forward and that they're not necessarily asking you like, what's the specific ROI of this? And that's something that I've been thinking about a lot lately, like when I have an idea for something and I want to go run to share it with our leadership team, like putting together the business case for that and explaining like, this is why I think this is a good idea. Here's the examples where this has come up or, and I know one other non-data scientist that I saw was listening in is our president here at Posit. And he's often one of the people that I go to with some of these ideas. So Tarif, I was wondering if you had anything you wanted to add to the conversation too.

It's interesting. I mean, I think both Joe and Ann raised really important points here. I like to sort of ask myself the question, like, what are the underlying assumptions behind anything that we're doing, right? Like whether it's on the strategy level, like what is the end goal here? And then how do I know that I'm on the right path? That being said, you also want to leave enough space for the moon shots, but I challenge everybody when they say, hey, I want to do X. I'm like, why? And what are the metrics that you are hoping to drive? It's interesting when you show the intake form, it's intriguing because that is an existing system that you have. The question that I find myself asking is like, okay, how long does it take before you know that you've got a good test? And when does the impact sort of kick in, right? Because some things, like in product development for us when we're building new features or new products or whatnot, it could take 18 months, 24 months before you know whether that investment was a good investment or not. But I think it's good discipline in general to sort of say, hey, you know what, here's what I'm assuming is going to change, or here's what I'm assuming is going to get better. Here's how I'm going to measure it. And then use that as a way to learn and get better, right? That's sort of my instinct on it.

The moon shot one is a good question. I think you might want to be okay with saying, hey, 5% to 10% of all the investments that I make are going to be these kinds of moon shots, and I'm okay if they fall flat on their face. JJ and I do that pretty regularly when we say, hey, you know what, we're going to take a crack at X or Y or Z and we'll make an investment that we're like, if it fails, it's not going to kill the company, but if it succeeds, could sort of generate a lot more revenue. So I don't know if that's what you were looking for from me, Rachel, but I push Rachel pretty hard when she's like, I've got seven ideas. I'm like, great, which one matters?

The rule of pi and time estimation

There was a comment earlier about your personal time investment, and it just kind of echoed me. So I used to build furniture. That was before I was in tech. And estimating how long a new piece of furniture was going to take was really hard. And so when I took really good notes, what I learned was this, that if I had built something similar before, I had to triple my estimate. And if I hadn't built anything similar before, I had to multiply times nine. And people, friends would say like, oh, you're terrible at doing estimates. And I'd be like, no, I'm awesome at doing estimates. And then when I got to grad school, I found that the mechanical engineers had names for this. They called it the rule of pi, that everything takes 3.14 times longer than you remember it taking.

And so the point of all this was just like, as you think about your own time investment, you've done all the ROI, but just making sure you have bandwidth to see this through successfully. If you've done something really similar before, triple it. If you've never done something similar before, multiply times nine. And I can tell you on my own project, usually just that ninefold thing is just like all the stuff you didn't think of. Like, okay, you put the model into production. Well, now you need to train every relevant user. You need to build enormous amount of documentation, enormous amount of tracking. This is all the stuff that was not part of the model development. And if you don't have time for it, well, your life's going to be really miserable for the rest of the year while you're working nights and weekends to see it through, or just budget, really over budget for time. And you can have a happy life and get things going.

the rule of pi, that everything takes 3.14 times longer than you remember it taking.

I just want to add one more thing to what you just said, Joe. I love this. I'm not going to think about this pi squared idea. But the other thing that I think is really important is to just remember that the more things you take on simultaneously, the more your work and progress increases, the more your context switching, and things will actually take longer than you even realize. I know this is kind of like meat and potatoes for a lot of people, but it's so easy to get sucked in and say, oh, this is not going to take long. I'll do this. I'll do this. Before you know it, you're doing 17 things at the same time. And that just slows the living daylights out of you.

Presenting ROI to leadership

This hour goes by very quickly here. But I know if people have to run at the top of the hour, that's okay. Thank you for joining us. But I'm going to stay on and hang out if people want to keep talking, too. But Brian, I saw you had your hand raised, too, and if you want to jump in. Yeah. So, I mean, I come at this. So, early on in my career, I was actually in charge of doing projects like this and actually presenting them as budget items for the Coast Guard to Congress. And you get your ass grilled.

And so, like, for instance, when I ran a data science team at an insurance company, all of our projects were ROI. The goal of this project is to shape X percent of this cost. It's worth X. Here are all the variables. These are what we think the ranges are. This is the project we should do. And the way we engage our team, to Therese's point, yes, when you multitask, you're really in trouble. We always had a, every team member had a short project, the quick and easy, the medium project, and the long project. Because in a long project, which means there is going to be time for stakeholders to make decisions and do stuff, and they don't turn around on a dime. So, you have a midpoint and you do a presentation, you're not going to hear anything for feedback two or three weeks. That's when you can work on other stuff. On a financial basis, I always just do stuff as what we call real options. Your first is your exploratory POC. What's that going to cost? What's the benefit? Benefit's probably close to zero, but it's going to be an investment cost. But then you have an option to kill, an option to expand, or an option to do something else. And if you can think about some of those options, you can price them out. And so, I had to present all my projects, five years, here's the return, here's the discounted value, here's the return, because our, I had to go through this CIO who went to the CFO, and that's how they looked at things.

You know, I want to buy Posit Connect, and it's cost me X amount of bucks a year for a license, these are the benefits I'm going to get from it, these are the amount of people that I'm going to have, it's going to cost me X amount of person, all these things. And so, I'd have these presentations to people, and you could go through those discussions, and we would do variable sensitivity. It goes back to that never present a point estimate. You know, this, it's a range between this and that, it's a range between this and that, it's a range between X and Y. And if you get really fancy, you can actually convolve them all in a simulation and say, here is the entire range of outcomes, and this is what it looks like. No, it's not a bell curve, it never is.

But, so, I mean, when you talk about ROI, that's how I've always thought about it, that's how I've always been forced to present it though, because if I'm asking my division leader, president, whoever, to spend 50, 75, or 100 grand, or to dedicate, you know, you think about a man, a person hours, 2,000 hours a year, maybe I want to dedicate 1,000 hours of the data scientists that cost me X amount of money, what am I going to get for that? And that's just some of the ways that I've had to present it and pitch it and, you know, get people to sign off and say, yes, we will give you the money, or we will give you the time to do that.

And to the earlier point, it's all about what metrics, what are you going to improve? You're either going to improve revenue, or cut costs, or improve efficiency, or cut costs. You have to have that in mind, otherwise, you know, it's just R&D, and if it's just R&D, then you've got to be able to pitch that to, hey, this is R&D, we don't have anything to work, but we're going in a space that we've never been before, and sometimes that's just, you've got to be honest with that.

Model improvement and ROI in insurance

Yeah, well, maybe while we're on the insurance example here, jump over to Adam's question in the chat, if you're still on, Adam. Oh, yeah, this could be a real rabbit hole, so we're almost out of time, I don't want to keep anybody, but yeah, I think the question is, you know, we were talking earlier about this idea of false positives and true positives, and you can kind of like, in a classification problem, you can kind of like tune that, and you kind of associate costs and things to those various outcomes. Yeah, so in insurance, I'm in property and casualty insurance, and we work on models that go into pricing, and it's way more complicated than just sort of like true and false positives. We sort of build a pricing model, we try to estimate losses for a given set of features, and then what happens is that all that information kind of goes over the wall to actuarial partners, our product partners who have certain kinds of business objectives they have to hit, so actually the models kind of get like tweaked and modified, and then they layer in additional things like discounts and so forth, and so when I go in thinking about, okay, I want to improve these models, maybe I'm bringing in a new third-party data source, for example, I want to figure out if this is a worthwhile data source to include. I have to figure out, is the model improving, but then I have to know also, is it worth buying, right, and so the models kind of decoupled in some ways from the final outcomes in really complex ways, and I'm wondering if there's some way to think about this fuzzy sort of connection between model improvement and ROI at the end of the day. It's a long-winded explanation, but...

No, no, I think a lot of decisions get made on overly simple metrics just because it's so overwhelming, like not all customers produce the same amount of value, but so many business decisions are based exclusively on conversion and retention, as though every customer were equally valuable, so but you're going to overwhelm your stakeholders if you're simultaneously like introducing like a new justification for like, oh, well, you know, deduct these costs, add these like lifetime value benefits, etc. Exactly. Those have to become separate issues, so then you just have this like reliable lifetime value metric you could pull off the shelf, and you could ask now, hey, what if we made decisions based on three-year revenue per customer rather than raw conversion, which treats everyone as equally valuable? You know, how much more money would we save? How much more money would we make? Those are all kind of your ways to shape ROI by redefining the question.

I don't know, maybe that was kind of the theme today. It was actually just like, you are in a very powerful position to redefine the question when you're doing an ROI investigation, and don't take that lightly, but keep in mind that people only have so much attention, so like you gotta modularize things so that they can stay on board. I see so many data science projects just completely overwhelm, you know, their stakeholders, so avoid that, and you know, yeah, you can get a lot of influence from that.

you are in a very powerful position to redefine the question when you're doing an ROI investigation, and don't take that lightly, but keep in mind that people only have so much attention, so like you gotta modularize things so that they can stay on board.

Budgeting time for moonshot and R&D work

Gerard, I know you had your hand raised, so I know we're right at time here, but I just wanted to see if you had time to ask your question. I just wanted to make a comment about these moonshot ideas and these R&D ideas. We usually, at the beginning of the year, just agree upon an amount of time that you have to do research and or for projects that don't necessarily have an immediate impact, and then basically throughout the year, if you come across a new really cool thing that you want to work on, the question immediately bounces back to you. Is this worth stopping the other research that you're doing right now and switch to the new project? Can you wrap it up? Can you switch? Because you only have so much time, and then there is not even a need to generate a return on that investment, and then usually I can myself answer the question whether I want to switch or not, because I either don't want to give up on the project that I'm working on or I'm done with the previous project. I hope that helps them.

Well, I don't want to keep you, Derek and Joe, too long, so thank you so much for sharing your insights and experience here. I just want to see if you had anything else you wanted to add to the conversation before we go. It's hard. Yeah, just give it 10 years, and it'll be a no-brainer. No, this was great. Great participation. Thanks, everybody. It was good.

Yeah, this was awesome. I would love to keep having these conversations, and I mean, I will definitely stick around here, but I understand people have to go to other meetings and can only spend so much time with us, but is this something we'd like to kind of

keep talking about or if there are other specific topics we want to do a deep dive on? Please let me know whether it's in the chat or if you have an idea later. You can message me on LinkedIn or email me. I'll just put my email in here, too.

Awesome. Thank you all so much. Yeah. Thanks, everybody. Thank you.

I know there were a few great questions that we didn't get to cover, so maybe I'll go and collect some of those, but Derek, if you are able to stay on, I saw one of the questions was about the infrastructure costs, I believe. I think David Sluder asked this, maybe. Oh, interesting. How does production resource costs, like cloud computing, factor into your ROI calculation?

Infrastructure costs and cloud computing

Here's an easy answer on the cloud computing. Given that we're in a hospital and in Canada with very particular regulations, we don't use the cloud. So the infrastructure stuff becomes part of operational budgets that we talk about in the beginning of the year with everyone else, including IT and other folks of like, yeah, if you want these projects to keep going, this is where you get to point to your previous ROI. Do you want projects that will continue to bring down costs and bring down readmissions and increase patient flow? This is the infrastructure we need. These are the subscriptions we need. Fortunately or unfortunately, we're entirely on prem, so I don't need to talk about cloud. For anybody who decided to hang around here, do you have any other last pressing questions?

Joseph, or apologies for mispronouncing your name, I saw you shared a framework in the chat and was wondering if you'd be open to sharing a little bit about that.

And no worries if I'm putting anybody on the spot. I'll try and go through the chat and pull out some of those resources there. Awesome. Well, thank you all so much for joining today. This was a really fun, lively conversation. I really appreciate it. Thanks again. Have a great rest of the day, everybody. I'll see you later. Bye.