Resources
Mar 1, 2022 |
1h 6min
|
604 views
|

Data Science Hangout | Matthias Mueller, Campaign Monitor | Understanding Customer Actions

We were recently joined by Matthias Mueller, Senior Director of Analytics at Campaign Monitor.

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
► Data Science Hangout site: rstudio.com/data-science-hangout
► Add the Data Science Hangout to your calendar: https://www.addevent.com/event/Qv9211919

Follow Us Here:
Website: https://www.posit.co
LinkedIn:https://www.linkedin.com/company/posit
Twitter: https://twitter.com/posit

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

All right, welcome everybody. So Rachel is taking a well-deserved break from work this week, and I am stepping in her place to host the Data Science Hangout. If you are a repeat attendee, welcome back. If this is your first Data Science Hangout, welcome. So glad that you're here.

This session is really meant to be for you. So I'll ask a few canned questions to start, but we want this to be audience-led, so please don't be shy and ask questions. You can participate in the Hangout through a multiple of ways. You can have your video and audio on and jump in. You can raise your hand. You can post a question in the Zoom chat, and we'll read it out loud. You can put a star next to the question when you ask it if you want me to read it out loud instead of participating. And then we also have an anonymous link for you to ask questions if you want to ask anonymously.

With that said, just in the interest of time, I figure I'll hand it over to you, Matthias. Maybe let everyone know who you are and what you're doing and why you're here. Yeah, absolutely. Well, thank you for hosting as the interim host. I appreciate it.

A little bit about me, my name is Matthias. I head the marketing analytics function at a company called CM Group. We're a marketing technology company. We're at this point, I think, the largest family of products and brands available in the email and personalization space. We're actually closing on another acquisition, I think, yes, as of Tuesday, folding Cheetah Digital into our fold as well.

We have roughly 70,000 customers across the globe and essentially utilize or we facilitate several billions of customer connections every year. My team specifically sits on the marketing analytics side. So there's groups that reported to me specifically tasked with identifying opportunities of how we can increase the revenue streams or reduce costs of acquisition.

It's basically split into two groups. So there is an insights and reporting function that deals with a lot of the questions of like, you know, what did we do and why did we do it and how is it performing and those types of questions. And then there's a more advanced analytics practice where we build anything from lifetime value, lifetime value calculation models to multi-touch attribution, marketing mixed models and those kinds of things, all with the purpose of supporting our roughly 100 percent marketing team in order to make good decisions.

I guess I'll stop it here because I'm sure we're going to talk about some of this in more detail throughout these questions. But that's the quick who am I and what do I do?

Very good. I mean, maybe for those who are not familiar with marketing analytics, give a little more insight with that or some of the data products that you can share that you're producing and the stakeholders. Yeah, absolutely. Great question. So I kind of touched on this a little bit.

So ultimately, the idea of marketing analytics is, can we try to understand how our prospects are acting on different properties? So whether that's a website, whether that's interacting with ad properties that we have, you know, that could be anything from, you know, simple Google paid search ads that could be OTT properties. So out of, you know, wherever we're buying ads, we're trying to figure out where do we get the best marginal return on our ad spend?

So with several brands and products that we obviously oversee, it always becomes a bit of a prioritization of like, you know, where is the best investment? Where do we feel like we can put the next dollar in order to get the best return in order to get more customers for less cost of acquisition?

That is sort of like the summary of it. But with that, there's certain or a lot of different levels of sophistication. So there is certainly a reporting aspect into it, like, you know, how from a marketing perspective and a lot of SaaS companies, the idea is to drive pipelines. So how can we drive, essentially, opportunity for our sales team to drive additional revenue into the company?

The way that we do this is through many different avenues from reporting all the way to, you know, more sophisticated analytical models.

Very good. And for anyone who joined late, I just want to remind you, you can ask a question anonymously. Tyler, my colleague, has shared that link.

I have not had a chance to read the question, but I see, Zach, you posted a question. Do you want to ask? Yeah, so again, so I presume GDPR affects America with UK legislation, how global everything is. So there's more going on to that. So recently, of course, we saw we've seen the whole Apple versus Facebook, Apple with their privacy changes, making apps have to explicitly ask for to track your data and everything. And of course, and everyone's anti-Facebook and all that data tracking. How do you see that changing the future of marketing?

Because, of course, the expectation is that legislation will be in place globally, the US will likely have one, the EU probably have another one that's probably more strict with everything that they do. And do you think that's going to impact how people not only go to market, but how you focus on different aspects of marketing?

Yeah, fantastic question. And you're actually spot on. So for those of you that aren't in marketing or have data protection laws to deal with, GDPR is a law that was written in 2018 in Europe, essentially governing who can access your data, who can process it, where it's processed and a lot of other things, really putting the ownership of data back into where it honestly truly should belong, right? Like the data should belong to each individual and giving a bit more options to how you can allow companies or disallow companies to utilize your data.

There's also US-based regulations. So there's the CCPA, so the California Agreement, essentially, which isn't as stringent as the GDPR, but it's kind of going into that direction. And I would certainly expect that there's more regulation coming. I know very recently there was a thing about, I guess, Facebook saying that they would pull out of Europe. I think that just happens in the last few days. And then the European foreign minister is saying like, OK, well, then do it. So I think there's a lot of back and forth that happens all the time.

But to actually get to your actual question, so I feel like marketing companies or companies as a whole shouldn't necessarily try to figure out how to skirt those different things. But there's a lot of things that can be done to actually test your marketing initiatives without being sort of on the creepy marketing side. There is probabilistic marketing attribution, for example.

A lot of the things that we've been doing internally is relying a lot more on testing and experimentation. So, you know, really interestingly, more traditional initiatives like building marketing mix models, for example, is probably going to become a little bit more invoking and because, you know, you aren't going to have that definitive click attribution that you are used to. We already see that if we look into channel attribution in different countries. So we operate globally and I can tell you that, you know, for US users, for example, we have a lot more insights into like their paths on our websites or, you know, what their different channel mixes are compared to someone like you that would be in the UK.

But yeah, ultimately, from a company perspective, it's about offering the right thing to someone. And I think sometimes, especially on marketing, we've gone to the point where everything and I say this as a data person, everything is so measured that people have forgotten that really, truly marketing should be about just offering someone or offering a prospect the best possible customer experience. And that's really what it's about. And the idea, hopefully, is that with that comes revenue that's attached to it.

everything is so measured that people have forgotten that really, truly marketing should be about just offering someone or offering a prospect the best possible customer experience.

And then from a data perspective, it becomes basically my problem to prove that, you know, there is a correlation between offering a good experience to actually having revenue driven into the company rather than saying like, hey, OK, we're going to, you know, we're going to cookie track everything because that's likely going to go away in the form that currently exists. Great question, though, I am and I could talk about this for a long time. Those are those are things that are always on, I think, anyone's mind that is revolve or working in the marketing space.

Thank you, and yeah, probably could, but probably let someone else answer, ask a load of questions as well. No, it's great. This is a good one.

The future of marketing analytics

Yeah, maybe adding to that, what do you think the future of marketing analytics looks like? Yeah, so I think we're going to be going into a I mean, everyone kind of in marketing talks about the sort of no cookie future. Right. I certainly think it's going to move in that direction where you're going to get less and less personally identifiable information on people.

From a marketing analytics perspective, I mentioned this a bit. I think it's going to come down to being more in the like testing and experimentation space. I think that that would be the one to still prove that value exists in marketing campaigns. And then, you know, there's different models, I think that would still apply on the probabilistic side to understand how marketing initiatives are going to be used. Except with the Web3 doorway, I don't know. Do you want to take it away?

Yeah, so this might be totally out there. It's something that's certainly been in the back of my mind with regard to if anybody's given thought about it, Matias, maybe you maybe you have as well. But I think it's going to be a really interesting thing to look at, and I think it's going to be a really interesting thing to look at, and I think it's going to be a really interesting thing if anybody's given thought about it. Matias, maybe you maybe you have as well, but just curious how if you guys are considering marketing analytics. It's not like Web2 is going away. But for how Web3 operates, are there considerations that you guys have maybe given for how you even do analytics in that in that kind of environment?

Truthfully, no, I'm not the right person to talk about that. Obviously, there there is a lot in that technology, you know, when it comes to blockchain and all of those things that are very interesting. We haven't necessarily gotten there yet from the analytics perspective, but I should probably note that down to try at least loosely thinking about it. But, yeah, I wouldn't be the right person. Is there anyone on this call that potentially has some background in that technology?

I do not, but I mean, I'm just wondering whether like all the information would be with one of those big tech companies and then we are sort of given like APIs to tap into those. Could be some of that is still already is already happening, right? Like a lot of the marketing analytics does happen through hooking into APIs of, you know, Google Ads and Google Analytics and all of those different things. So I think there's certainly an argument to be made for that.

Yeah, it could be.

Cool. Just curious, I figured if there was a time to ask, this would be great. No, it's great. Yeah, it's a great question. We haven't dabbled into that technology space for a little bit. But, yeah, I'll definitely take that away because I think that will be an interesting future proofing project as well.

Data scale and company structure

There's an anonymous question about the size of the data that you are analyzing or the type and the real question is, are you dealing with big amounts of data, digital touch points? Question mark. Yeah, so the quick answer is absolutely. We have a treasure trove of data. In fact, sometimes it feels a little bit like there's too much information and it kind of becomes more of a question of what is actually interesting rather than, you know, can we look at everything?

The longer answer is that we have a bit of a odd company set up in that we are essentially a parent company to different products that are all in them or different brands that are all in the marketing technology space. So each of those different brands that we own, so, you know, CM Group, the company I work for, owns the namesake company Campaign Monitor, but also Sailthrough, Seligent, Emma, Cheetah and a few more.

There is different levels of sophistication that we have in terms of data models for all of these different products. So the longer answer would be it depends on, you know, on which brand we're talking about here internally.

But to generalize, I think on the marketing side, we are probably the most measured function in our company. We tend to track and over track everything we can track.

And it's become the idea is really how can we find like those needles in the haystack to actually try to tell us what we should be looking into?

And one of the and I mentioned that my team is kind of split into two groups. There's like a insights and reporting team and then there's advanced analytics team. One function inside of the advanced analytics team is like an automation practice where essentially we write a lot of code that deals with anomaly detection.

And the idea behind that is if we have, I find a really bad metaphor, say like our data is the size of New York City, right? There is no way for us to actually look into that and kind of investigate the whole New York City. So through the automation anomaly detections that we write, we try to eliminate sort of like most of New York City and just tell us, OK, these are the specific houses that you should look into that have something unique going on.

And then the insights and reporting function that I have would then actually go into these individual houses and search for a specific item in a different room that I hope that metaphor holds true. But essentially to answer that question, I think it's more of a question of how can we identify what is actually important out of like that large treasure trove of information rather than not having enough information to work with?

Audience Q&A: insights, Slack bots, and business decisions

I see there's a hand up. And Jiwon, is that how you pronounce it? Yeah, that's right. Hi, Matias. I watched your lead anomaly detection talk and I implemented something on my own Slack bot. So first of all, thank you for that. I love it. Yeah, I'm just wondering how the insight finding leads to business decisions. So I'm kind of trying to balance out the granularity of the information that I share and I'm wondering how that conversation goes within your team.

OK, first of all, I love that. I love hearing everything about this. And we could probably talk another couple of hours about, you know, where I feel like business intelligence has a place in the business and where it does not. And what we found, again, we have this specific use case that my team is sitting on the marketing team. We're integrated into marketing and we have a lot of people that are the end users of or the stakeholders of our analyses that do not have like a quantitative background, right? Like you might be able to or you might have to do an analysis that has a regression in it and then explain it to a team of copywriters. That is a very unique thing that happens here all the time.

And what we found and we have a, I would say, decent stack. So we have a Looker instance sitting on top of a Redshift instance that flows almost all of our data from a reporting perspective flows into that. But what we found is that building more and more and more business intelligence didn't really increase the knowledge or data data aptitude for our for the teams, because there is a hesitancy of utilizing those BI tools. You know, if you don't if you don't have like a data background.

So instead, this is what what you're referencing there. So instead, we've actually kind of flipped it around. So we wrote this Slack bot that basically takes information, you know, in this case, like, you know, some anomaly with detection and then surfaces that information right to when it happens to people on Slack, just pinging them like, hey, something's going on in your area that you should be looking into.

So with that process, we've been able to really kind of change of how people digest analytics in that, you know, it doesn't it doesn't it's not like an active, oh, I'm going to go and look at data, but it's basically you're being furnished with data in the moment that it's interesting to you, hopefully, based off of my my opinion to to now get to your question, which is like, how do you not overwhelm people with insights and all of those things? It definitely is like a definitive or distinct sort of area that you have to walk the fine line between not, you know, shooting people and spamming people with information.

But what we found based on like the iterations that we went through with this thing. Is that as long as you're very precise with what someone wants and like understand what someone's exact requirements are for specific pieces of information. So, for example, our director of performance marketing, she wants to know every time when our cost per click falls below a certain threshold based off of where we would expect it to be or if any region performs out of the ordinary in terms of impressions or click volume. So when that happens, that's when she gets like a Slack notification that we we wrote, you know, hooking into a data warehouse and shooting it off with R and Slack.

So to hopefully answer your question, and I kind of like took a very long detour, so I hope I did. If not, please, please ask again. What we found is that the success of this ultimately comes down to how precise you are in understanding what someone actually wants to know and then giving that person that information. What we found less success at was to hook into like generic channels and just kind of having more broad insights pushed into these channels, because as soon as you do that and you do that on a recurring cadence, if it's not pertinent to someone, then they will stop reading it after a little while.

What we found is that the success of this ultimately comes down to how precise you are in understanding what someone actually wants to know and then giving that person that information.

So that was, I think, the one takeaway that we've had after sort of thinking about this a little bit more and getting some feedback from people that were exposed to it, that it really kind of depends on how interesting is this information to me and can it actually help my job be easier if I have this information?

Did that answer any of what you're saying? I know I kind of like went way out there. So definitely feel free to tell me that I didn't help with that question. Yeah, no, absolutely. Thanks so much. Good. Yeah, great.

Getting requirements and understanding stakeholder needs

How do you go about getting those initial requirements or getting feedback from people that this is what I want? This is kind of what I want, but it's not really what I want. How do you encourage people to be honest? Yeah. Oh, great.

I think it's more, first of all, I think of data products as living and breathing things. I think data products are not something that you publish and then let go. They're kind of like you're forever tied to it, especially if it's something that pushes information to people. So you're going to have to be open to feedback from people. Ultimately, we are, as analytics, we are a shared service that is a service provider in some shape to the greater marketing organization. So our success is measured by how well people adapt the data that we give them.

Because if we didn't, then we can have all this information if they don't actually use it to change their behavior, if they feel like they need to change the behavior based on a data point that they see, then why even give someone the information? So I guess that's on the back end. Everyone should always be open to making another version of it or getting to the next iteration. On the front end, though, how do we make sure that we understand what someone's requirement is?

I think the big thing that I found here is that it helps to try to kind of try to get to like the core of what someone's actually trying to answer rather than getting into like a solutioning process too early. So sometimes we have people that will come to us and say, like, hey, I would like to see I would like to get something sent to me that tells me how I'm tracking against the end of the month in terms of pipeline, just making this up. And I would like to see it by region and I would like to see it by marketing channel and I would like to every day. So it's like in that case, someone came to us already defined sort of like the report criteria that they would like to see.

And we've we can certainly build that. But what we've actually found over time is that doing that without actually understanding what their true question is, because they wouldn't come to you if they didn't have a question that they're trying to answer. Right. In this case, maybe they're trying to find out, you know, maybe, you know, in this case, you know, end of month pipeline could be something that they're trying to see how they're tracking against the target. You know, how how are they going to end up at the at the end of the month? So understanding more of like why they're actually interested in this kind of information and then trying to see whether that is truly the best way of giving that to them, like this specific already defined report.

I think a lot of it happens there where you really speak through, you know, let's let's put away what you have in mind for a second and let's understand what you're actually trying to answer. Like, what's your question? And then can we try to come up with a process that would give you, you know, or get you very close to getting all the information that you need in order to make a decision on, you know, whether you can answer it or not?

Chat questions: serving users vs. serving ourselves

Very good. I want to make sure that we get to the chat questions. I'm going to go in chronological order here. Rick, I see that you had a question. I don't know if you feel comfortable with jumping in. Yeah, sure. Thanks a lot.

Because really, I enjoyed your answer, especially to the anomaly detection and delivering the results with the Slack bot. I didn't think it was off topic. I think it was really spot on because it really struck me that it's kind of ironic because you're working in the marketing department and actually there was a marketing problem. You're trying to sell something to the people and you want to see that they're buying it basically. And I mean, the people that you're talking to would tell you the exact same advice that you're giving me too much information and it's not the right information and I'm going to ignore it. And as marketers, they would appreciate immediately that advertising in the wrong spot and advertising too much wrong information to the wrong people in the wrong spot. So it seems like it's all kind of connected in that.

And I feel that oftentimes we do this as data scientists. And I wonder if you agree. As scientists in general, I think we tend to service our needs because I spend a lot of time making a really fantastic pipeline or algorithm or look at this, how beautiful this model is. And they don't care. They just want to know the answer. Right. Like I don't I don't need to know why Google put this email in the spam folder. I just want to know that the junk is getting into spam folder, although they have some wonderful algorithms and back when it does that. Do you think that we oftentimes service our needs instead of the needs of our end users?

I completely agree with that. I mean, I think it's sort of a self-made problem on the data science end too, right? Like, I mean, if you're if you're trying to digest any kind of content that's written in the data science space, it's always content about like the newest new shiny thing that came out. Well, not sure that misnomer here, like not shiny, but like the new sort of, you know, the newest thing that came out, you know, a while back, everyone's talking about like GPT-3, like all of those things are like things that I think we've done a little bit to ourselves where we want to do like the cool stuff.

But, you know, data science or like let's say like machine learning might be a good example in that even where, you know, I think a lot of companies go out there and they now understand like, hey, we should probably hire data scientists because that's what people do now without, you know, ever paying attention to like, well, maybe there should be like a data engineering function first to actually build the house before we can hire like an interior decorator that would be the data scientist, you know, coming in and making everything pretty, like I actually have to build the walls of the house first. But they may just not even have the data that. Yeah, I think we do a lot of that. Yeah, I think you're spot on with that. I certainly do it. I try to be mindful of it.

So one thing that strikes me is that I feel that oftentimes they don't see how much work went into doing something when you do show them that result. And I feel that we constantly want to kind of up our game and show how much we're worth and that, OK, we're really adding something to the bottom line and we're really proving our worth because I think there's still some like, all right, like was it worth having this team and hiring these guys? So do you feel that there's ways to really show the value of your work without just I think we make it sound more complicated than it is to do that? Are there better ways than just overcomplicating our analyses and our presentations?

I, Ryan, I saw you unmute it, did you want to add anything? Yeah, I don't want to take over your talk, but like I think about this all the time, like like I've been part of my life forever and and I love that this has come up straight up because this is this is make or break for data science like it really is.

We will only get so high until we solve this problem. Right. So it's it's it's a great topic.

I think part of the piece that is part like that causes this problem are kind of two elements. One, if we're talking about the two different parties, the data science party and then the business party, we see different things as cool. Right. Like we just do like different. We have different values of what is cool. And you talk spot on about like what we think is cool, the shiny objects, the how that's not cool to them. If you use sales as an example, what's cool for them is making their commission. Like that's what's cool and that's what speaks volume. Right. So understanding the language to talk about is part of it.

But how do you get there is what we're talking about. And it's probably one of the most under discussed, in my opinion, boring. More challenging and not really start up a bowl so you don't get a lot of the financial attention is data literacy. Right. That's just straight up literacy. And I'm not talking about like, OK, how what's the syntax of writing something and how to build? I mean, what can I do for you just at that core level? What can data and analytics do for you other than increase your shareholder value for VCs? Like, like, really, what can it do?

And we have this potential problems for most educational systems is data science has gotten us really well at certain levels from mathematics, engineering, comp sci, stuff like that. But no one else is learning this kind of concept. So there's the salt or oil and water kind of piece. And we in our education systems, we're not really bringing that cross pollination between the two parties. So how do we get there?

Like, Rick, I'm having that right now with marketing, with HR and all those pieces, and it's bringing them along the journey and like forcing them to write stories. No, it sounds silly, but sorry for all the emails that are pinging up in the background to write stories. And in that time of gathering requirements and turning it in fun, I educate them. This is what that means. That's what it is. So they see like, holy. There is a bunch of stuff happening, but that's why you can't get it to me in a day. That's why it takes you so long to refine it. And like they kind of learn along the way, not that I want them to have a Ph.D. in machine learning or anything. I want to be like, yeah, that's a question that they could ask. No, we will never get there. That's not we're not advanced as a society.

No, I took over your time. So I'll just be quiet. Oh, no, no. I love the I love that. I think there's two things I'd add to this, too. Was the first thing and this is why I actually like Robert's question so much. The first thing, truly, it comes down to understanding what the requirements are on a core level more than, you know, this is the data I'm looking for. But like, what are you actually trying to answer and how does it impact? You know, like you said, for a salesperson, ultimately, you know, how can this data product that you're building for me help me actually make more money? Right. Like that's what's what it's after.

But then, Rick, the second part is that I think one thing that we sometimes do on the data science and is sort of disappear in the process. Right. You you do something. You have you have your eyes on like, oh, this I know now how to do this. I know now what I want to build. And then you data science does take a long time. And I think it's really, really important, like Ryan said, to take someone on the journey with you and kind of like let them know what is happening throughout.

We build things internally. It takes several months of building like a marketing mix model, for example, is something that is probably like a six to eight months project. If you don't take someone on that journey with you, they think you disappeared for six months. Right. So it becomes incredibly important to to speak through and have a very clear understanding of like, look, this is like all the things that have to happen. These are the things that we're going to need your input for and kind of like make someone feel that they have ownership over that project. Even though you're the one or your team is the one writing code. You're producing that project or data product for someone else, not yourself. And I think that's what this comes down to.

You're producing that project or data product for someone else, not yourself. And I think that's what this comes down to.

Yeah, we were we were chatting at RStudio with Uli. He was previously the chief data officer at City of Reykjavik, and he was saying that when he starts a project with a stakeholder, he likes to go back quickly to them a few days, a few weeks and show like, here's what we here's what we can do. Here's what we have. It's not finished yet, obviously, but like this is what we're thinking about. And his perspective is that makes it real to them. It makes it valuable to them. So what you said made me think of that.

I see, Javier, you have your hand raised.

Yeah, thanks, I was just going to add to some of the comments and maybe to Rick's question. One of the things that I feel like I did wrong in my last job was make the end deliverable. I mean, in my case, it was a shiny app, but is make it way too like geared for someone that has analytics training. I mean, I had like a FAQ page that explained a lot of the different visuals or numbers that you're seeing. I thought it was great because, you know, my perspective was, hey, like I have all the instructions here, like someone that is a non-technical you know, salesperson or account executive. They can simply look at this FAQ and understand what's happening. But it became clear that like the data I was trying to communicate was not intuitive.

And so through a few iterations of like that app, as well as a few new apps, it like the way I tried to start building them was to have no instructions. And that really kind of helped communicate all of our data points or visuals a lot better. Like someone who's non-technical should be able to go into this dashboard and kind of intuitively understand what it is that you're trying to analyze or, you know, the actionable items. You shouldn't need to give them training over training or a long FAQ in order for them to like come out with actionable insights. So anyway, just wanted to chime in with that.

Data translation and product adoption

I love that. You know, I mean, what's ultimately our purpose being in analytics, right? It's it's not to produce like dashboards and give people data, but it's to make people understand what the data means. And I think there is a very definitive difference between the two, you know, which is also one of the drives why we actually went away from just churning out dashboard after dashboard, because you don't actually know how someone receives that information if they're on their own.

So, you know, I think over the like the last few months, years, sort of like the concept of data translation has been like a little bit more sort of up and front center. I don't know if anyone has like heard of that, but like, you know, Google is now hiring like data translators, which, you know, feels like that's kind of what we should be doing, I guess. But like that's what it comes down to. And I think you're on the right path where, you know, if you're building a data product, I think the idea is that you have to build it in a way that it's intuitive enough for someone to use it, because if it's not, then it can become it can start feeling like work. And you don't I don't think that's the purpose of us, right? We're as as people in analytics or data science. It's our job to, you know, give people more information and help them make decisions based off of information that we can give them, rather than kind of like adding another thing onto like that plate that they have to check every day. So I love that you went through those iterations and, you know, kind of like thought through like what was working and what wasn't. That's that's great.

Mike, I see you have a hand raised. Yeah, sorry if I'm off beam here, but isn't this essentially a kind of agile approach, you know, because if you if you listen to what they want, right, and then you're interpreting what they're asking for in terms of what you can actually deliver to them. But if you know, if if it takes you six months to get an answer, as you've just said, Mattias, they're not there. Essentially, you're not there for them in that time. But, you know, they better not wait six months to find out that they could have got the same answer through a normal regression within six minutes. So you then got to kind of go, well, how do I keep them on the hook and bring them, as you say, bring them with you, but keep delivering stuff to them that will say in the absence of my all encompassing model of absolutely everything, let me give you something just now that will help you and convince you that it's worth waiting six months. Is that fair or am I?

I like that idea of thinking of that way. I think ultimately it comes it comes down to the ownership idea where, you know, data products are for someone. Like we're not building data products for ourselves in mobile. We're building data products for ourselves in most cases that I guess there might be instances where, you know, like we have internal stuff running that just checks, you know, whether certain ETL processes ran correctly. But in most instances, a data product in the end, while we're the ones that know how to operate it, how to troubleshoot it if something goes wrong, how to build the next iteration off of it. They're not our dashboards or they're not our, you know, models, they're there for someone else, and I think that's where it comes down to where you need to have people feel that they own this thing rather than, oh, this is something analytics did. I don't understand it. It just does something and I use it.

They you know, that's that's where I think you get the adoption idea from. And yeah, I'm a proponent of the idea that, you know, you can really only build a product for someone if you involve them in the process of when it's being built. So, yeah, I agree with that. We kind of use a sort of faux agile methodology internally because the marketing our marketing team isn't on like we have a project, a product project management tool, but it's not on the technical side. So like it's not like we use Jira or anything where we could do a true sprinting process, which, again, is kind of like us trying to adapt into the marketing organization and trying to use what they're using, even though, you know, it's it's not the perfect solution for us. But I think that's where it comes down to. We try to be as helpful to the rest of the organization as we can be. Whatever that might mean.

Yeah. A wise old statistician once said to me, people are going to make decisions whether you're ready or not. You know, so so you'd best have something that will at least influence them in kind of the right direction. And as I say, like being able to at least say it's worth waiting because, you know, we can give you something good now, but by golly, by the time you really got to the bottom of this, it'll be worth waiting for. I love that. Yeah, I subscribe to that.

Outlier detection and emerging patterns

In the spirit of remaining true to the format, Zach, is the next question I have time stamped around outlier detection. Yeah, so for me, like I find it quite interesting looking at outliers becoming patterns, especially as you get more and more data. Of course, I'm guessing for you, Matthias, you have already loads of data. So it's not like you're getting lots of data, but patterns do emerge outliers. You I don't know.

My general idea at the moment is if it happens once and it's an outlier, if it happens twice and I research a bit more and it happens a third time, then I go, actually, this isn't an outlier. This is becoming an emerging pattern. Of course, like scales one, two and three depend on how big your data you're looking at. I was wondering how you do that with the size of data that you have. I'm assuming it's all automated.

It is incredibly interesting to talk about this question, because if you had asked me, you know, I think I've been with CM now for like roughly three and a half, four years or so. If you'd asked me before my time here, I would have given you a completely different answer, because I think from a data perspective, the first thing that you generally do, you know, like the first thing that they teach you is, well, just, you know, the outliers probably should be normalized out. Or, you know, like if you build a model like, you know, like you kind of have to make sure that it's not affected by outliers.

I've actually completely like changed my thinking on that working here, because some of the products that we have are like the outliers are the things that matter. Like the specifically for a brand like Sailthru, for example, which is sort of on our enterprise and, you know, the marketing there is on the ABM side. So what does this mean? It's not a volume business, but it's, you know, very much, we essentially already kind of know the opportunities, the accounts that we'd like to target, and we target that way. And, you know, it really makes a big difference if a singular account like closes or don't, or doesn't. And, you know, from that perspective, I've changed my stance on outliers,

because it feels like for some brands, those are the things that are actually interesting. And when it comes to outlier detection, that is something that you have to keep in mind that is it for something where it can be like a something that you're actually looking for? Or is it something that you're trying to figure out how do I build a good model so, you know, it still is statistically valid.

There is a lot of, I forget who said the whole of this is startupy. But like, there is a lot of startups now that deal with this type of issue as well. Like one big one is like outlier AI. I don't know if anyone's heard of them. But they do similar processes where you essentially just push all the data that you have to them and they build models for you on the outlier detection.

I felt like I want to be the owner of that. So we do, we write all this internally, because of this reason that I don't want someone else to make the decision of this specific outlier. Is this just, is it a true outlier that, you know, a couple years ago, I'd be like, okay, let me just ignore this. Or is this something that we actually should look into further, because there's a business opportunity here. And in many cases, we found that that is actually what's happening.

So, yeah, outliers are a finicky thing. And I feel like I kind of go back and forth between, like, what I think of them, you know, depending on what type of project I'm working on, whether, you know, I love them one day, and then, you know, I hate them on the next because they mess everything up.

Educating the broader marketing team

Okay, thank you. Santiago, Santiago has a great question that he put into the chat. All of them are great. I have to say that, but I particularly like Santiago's. Are you still on the line, Santiago? Would you be comfortable sharing?

I am. So let's go back to kind of the previous line of thread of questions. So outside of a specific project, and educating folks on that, have you done anything successfully or failed even to try to educate the broader marketing team? For example, have considered Lunch and Learns, or like a newsletter or something? I don't know. Any thoughts?

Oh, goodness. I think it's, if you figure out how to do it, it's the Holy Grail. In a sense, it is a bit about setting expectations, right? Like, what does democratizing data actually mean? And if you set out for, you know, like I said, we're, you know, a bit of 100 people in marketing, setting out with the expectation that everyone's going to be like a closet data analyst, that's not going to happen.

However, what you can do is make people understand the power that data has. And the way that you do that is to give them the understanding, or have them understand how well it can help them make decisions. I think in this process, one of the things that I try to emphasize is that I'm not the decision maker on things, nor is my team. So when we do analytics, and we provide like a data product to someone, it is still that someone's decision to make in the end of what they want to do.

I don't take that power away from my team or from the marketing organization. And I'm very protective over the fact that I feel like data should be a neutral party to someone, like a completely, essentially just giving someone, here's everything that we feel like from a data perspective, you need in order to make a decision. But then the way that you empower someone, and actually have them understand what the value of information is, is if you still let them come to their own conclusion about how this decision should be made.

And I'm very protective over the fact that I feel like data should be a neutral party to someone, like a completely, essentially just giving someone, here's everything that we feel like from a data perspective, you need in order to make a decision.

I think some organizations have the approach that, you know, there's like a, you know, maybe a business, a biz ops team that does all the analysis. And they, you know, push essentially their thoughts into like an organization. And then that's the thing that everyone kind of has to follow. And I think that's a really, really quick way of having people sort of not participate in this idea of, hey, data is actually important. Because you can give people data, but the only way if that people will actually start using it and that you truly democratize it, has to come from that person. It's not something that you can force someone to do.

So that, I guess, is my longer answer to this. We've certainly done a lot. In fact, I, next week, I'm teaching the entire marketing organization, Looker, yet again. So I'm still doing sort of lunch and darns. Absolutely, I don't think we're all going to get around that. But yeah, the idea is people are going to, to me, people are going to adopt it if you make them understand that what they can do with it can actually make their life easier. And it can help them sort of put more behind their own job, rather than, you know, kind of mandating, hey, this is the data that you need to use. And this is the right way to do it.

I think it takes a little bit more of a defined approach to make sure that people actually buy into your idea of being a data-driven organization.

Cool. Yeah, that all makes sense. I haven't started this. And should I ever start this, I have to pitch this idea and get by and you know how that goes. Maybe I'll report out someday.

Thank you. It comes and goes in waves. So yeah, just keep with it. It's worth it once you get there.

Proving value and monitoring progress

Morris, you've been patient. Thank you for that. But I see you've had a string of a few questions. Welcome aboard. Yeah, thanks. Thanks very much. My first chance to speak. Mattheus, this has really been a helpful conversation for me. I really appreciate this. My context is totally different.

I work in a non-profit area and most of the people that are working in analytics, at least in my part of the organization, got here through some internal mobility. We are the closet analysis analytics people. And I think that's probably true for a lot of us in this field.

So I am looking at ways to monitor progress in my just gaining relevance for all of this. I've gone through a lot of iterations of trying to convince people that something's a really smart idea or smart models, but it hasn't worked. So I sense you seem to be successful in what you're doing. How do you monitor that? How do you know that you're heading in the right direction? What are the bright spots you look for?

Yeah, I love that question. And also, I love that idea of having a non-traditional path into analytics. My undergrad is in visual and motion design. So I definitely have a very weird path into analytics. So I'm on your side on this.

To answer your actual question, though, to me, it comes back to, can you measure how successful this actually is? And how do you do this is through testing and experimentation. I think the way to prove that something is working or not working, the quickest way to do this is if you think about statistics, can we understand what would have happened if we didn't do it? Or can we understand what's changing based off of our proposed changes?

Obviously, I don't know exactly what you said you're in the nonprofit space. So obviously, not entirely sure where exactly that would be applied. But from a marketing perspective, for example, we do a lot of when we propose an action. So say we find that we are running a podcast ad, or say the marketing campaign manager for a certain brand wants to buy a podcast. And we would likely suggest from an analytics perspective, then when that podcast is bought, can you buy it in certain DMAs? So DMAs are marketing regions, essentially, where then we could track, OK, what's happening between all the regions where the podcast was run versus the one where nothing happened? And then is there an actual lift in that?

Obviously, that's a very sort of marketing way of thinking about it. But I would suspect that there is a lot of things that if you think about what would happen if nothing, if we didn't do anything, and you can keep a hold out of that, and then compare to what's actually happening, that would be a way to actually get definitive data on whether or not whatever you're doing is working well. Does that answer your question? Or was it more from an organizational perspective? And now as I was talking through it, are you trying to prove value for the organization? Are you trying to prove value for whatever you're actually doing?

Yeah, it's interesting. I think it's the latter. I think there is the mission of the organization, what we're trying to accomplish. But then also, part of that is internal development. My most recent mandate is to help us become more data-driven. So nothing too trivial. So it's both. It's both and. You know, I have a great vision of being able or trying to help us to, like I say, accomplish our mission. But I think internally, there's this idea that we really have to shepherd the resources that we have well. And how do we do that best internally by making the best decisions? So it's decision-making in that.

It's interesting because becoming data-driven is obviously very bust for me. I think if that's a definitive goal, which is fantastic, I think one of the questions I would have, if this is like one thing that, you know, look, Morris, your job this year is going to be to make us more data-driven, to go back to trying to understand what the actual idea is behind what they're trying to accomplish. I think I would have asked, you know, sort of, I'll preface this with the fact that I work in a very non-corporate environment. But if I think about like my corporate days back in like older jobs, I think one of the things that stuck with me are sort of like the idea of smart goals. So essentially, you know, can you measure your goal? And being data-driven is really hard to measure, right? So like I think if someone gave me that role or that idea or that quest, you know, how can you increase data literacy or, you know, make people more data-driven? I think I would want to know what does that actually mean to the person that's asking me what I've done.

I can tell you. So weirdly, I've been in a similar boat in the past where, you know, I just mentioned Looker trainings. So this is sort of like a, you know, hit to the past for me, but somehow I'm still responsible for teaching everyone that. But what we've done in the past several years ago, one of the things that was on my sort of, you know, list of responsibilities was democratizing data across the organization, which is similarly open-ended. And the way that I tracked it and the way that I could prove it for these Looker trainings, for example, I had set out a quick survey before the Looker training, gauging everyone's understanding of what Looker is, what their level of comfort is, can they build a dashboard and manipulate a dashboard, you know, and kind of like understand where the baseline is and then did the training. And then two weeks later, essentially asked them again for a survey, which did show like pretty considerable increases in like how people felt about how they could use business intelligence.

So that might be in a way for you if you have to sort of get buy-in from leadership. Those were the things I would think about, like, how can we prove that whatever we're doing, you know, has some success with it. And yeah, that's how I did it from the BI side. And yeah, there's probably opportunities like that.

Yeah, thank you. That's helpful.

Balancing new technology with current skills

Do you have time for just one last question, Matias? I want to make sure that everyone who put a question in the chat gets it answered. I don't think Peter is on any longer because he said thanks all, but so sorry if I'm reading your question wrong, Peter, but feel free to jump in. His question was, how do you balance steering clear of flashy new things when keeping your skills current? I don't know if I can.

So it definitely is a balance. I think that's where it comes down to because I think technology is moving so fast that if you don't look for those shiny things and at least know about them and kind of have an understanding of what they do, then it can be rough, right? Because who knows what's going to be around in five years. I think one of the good examples, and I'm going to really completely butcher this, but

if you think about like book printing, right, that was like, this is where I butcher it, maybe 1400, 1500 or something. And then, you know, like the actual first digital printer was like in 1990. And then all of a sudden, you know, a 3D printer was in like 2000 and whatever. So I think that is the pace of technology where, you know, you go from a very sort of study, you know, innovation to like very, becoming very exponential very, very quickly. And I think the same is true for data science. The same is true for analytics. There's so many things that are happening all the time.

See, I feel like I was doing well on the 1400 to 1500. I got to pat myself on the back there. But technology is just moving super quickly. And if you don't have an aptitude to kind of like follow things and kind of be open to learning new things, then I'm not sure if analytics is going to be a great fit for you because there's just so many things you're going to have to keep up with.

On the other hand, and I forget who said it, but ultimately, what we talked about earlier, you have to keep in mind that what you're making is only as good as, you know, people understanding and people utilizing it. So sometimes a more simple model that's more easily read and more adoptable outperforms the shiny thing.

So sometimes a more simple model that's more easily read and more adoptable outperforms the shiny thing.

So I think in that sense, it's really important to think about the business requirements. So you cannot sort of let yourself make that mistake of, okay, I just really want to do natural language processing because I feel like it would be really cool if it has no application to what you're doing. So you can't let your sort of personality try to dictate what the business needs. I hope that answers it.

Very good. I'll speak for Peter and say, great answer. Very good.

Closing thoughts and finding Matthias

To wrap things up, I thought maybe one, how can people find you? And then two, any sort of imparting wisdom to people who are interested in getting into data science leadership, moving up in their career. I mean, it's really fascinating since we started doing this, it's like almost like a hundred people join every week and sort of all over the career grid. So I thought I'd ask that because there's a wide range of audience here.

Yeah, I'm a huge fan of these talks. It's one of the recurring meetings that I don't dread. You have these days with everyone working from home, it seems like there's just meetings and meetings and meetings. And in many cases, dedicating like a whole hour every week is something that can be challenging. But every time that I've logged into one of these, I've just found incredible value. So thank you. I guess I'll give that right back to you. Thank you for putting this on. They've just been like really eye-opening.

How can you find me? I'm on LinkedIn. That's probably the easiest way. Otherwise, yeah, LinkedIn is probably the best. I don't like, I'm not very selective. So if you send me an invitation, I'm sure I'm going to invite or I'm sure I'm going to accept it. Other than that, yeah, I guess email me or find me through a pigeon. But I'm happy to connect with anyone that has additional questions about this or just wants to chat, data science.

And the last question that you had was about like how to actually get into data science and data science leadership. I don't know if I feel like a bit pretentious talking about this and answering this because I don't, you know, I certainly still feel like I have a lot to learn to move into that realm also. But in a sense, I think to me, it's understanding how you can provide business value to someone and really kind of trying to be or trying to use your skill sets in order to make someone else's life easier. And once you show that value, there's natural buy-in that generally people will help you along the way. I, yeah, I guess that's what I would say.

But yeah, I don't know if I'm like, there's been some incredible people on this, you know, that probably have a better pedigree of answering this in terms of building like a huge data science team on their, within their companies. I think your answer is perfect.

Great. Well, Robert, thank you so much. That was fun. Absolutely. I mean, thank you. All I had to do was tee up a few questions. Thanks everyone for participating. As always, next week, same time, same place, forever and ever until we run out of data scientists.