image: thumbnail.jpg
Josh King @ The Knot Worldwide | Creating your data and analytics roadmap | Data Science Hangout
At 36:28, we discussed: What goes into creating your data and analytics roadmap?
I'm actually in the midst of this right now– as in the hour before this meeting.
One of the things I like to come back to is from the CTO at my last company. He had this really good framework for thinking through this, and it's to break it out into 4 things:
1. Your mission
2. Your vision
3. Your maturity model
4. Your strategy
If you want to look it up, his name is Dave Kaplan and he has a great article on Medium: https://medium.com/policygenius-stories/missions-visions-maturity-models-strategies-ce5a4a408ab2
This gets into a good framework for building out and identifying the different aspects of a strategy that you should consider and having a clear alignment for what that will look like.
Generally, a mission is like a single sentence for why your team or your department or your company exists.
Your vision is what it means to succeed within that. So, if I'm thinking through marketing data science & analytics - I'm thinking what does it mean to have a best in class marketing data science organization across relevant brands?
Your maturity model is, where are you along that map from very basic early foundational building into “I am best in class”?
Then your strategy is how you get from where you are now to where you want to be in your vision. So, this really aligns a good framework for what it is that you're building out.
And to the other part of your question, does it cover all data work?
I have data scientists and analytics that report into me, but there's also some of a dotted line from all of our data engineers that focus on marketing problems so that can include BI, data operations, data warehouse, tracking architecture.
As we're thinking through a bit more holistically in terms of all of our data functions that are supporting marketing, I identify, what do we need to drive decision making for our marketing organization and not only how does data science and analytics fit into that, but what is the infrastructure we need from a data engineering perspective that would enable us to be able to do this work?
Yes, it's not necessarily my vision or my strategy that will get us there but working really closely with our Head of Data Platform so that my vision and his vision are really aligned together and we’re thinking through a bit more holistically as we're building that out.
How long is it? My current focus is on building out a 1 year, 3 year, and a 5 year roadmap to really have some intermediate stages along the way that we're getting towards.
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Transcript#
This transcript was generated automatically and may contain errors.
Hello, everybody. Welcome back to the Data Science Hangout. Hope everyone's having a great week. If we haven't had a chance to meet yet before I'm Rachel and I lead our pro community at Posit. So the Data Science Hangout is our open space to chat about data science leadership, questions you're facing and getting to hear about what's going on in the world of data across different industries. And so it happens every Thursday at the same time, same place. So if you are in the future on YouTube, and watching this recording, just know that we'd love to have you join us live too. And you can use the link to add it to your calendar in the comments. But together, we're all dedicated to making this a welcoming environment for everybody. And so we want to hear from everyone at the hangouts, no matter your level of experience or area of work. And so it's totally okay if you just want to listen in. Maybe you're eating lunch or going for a walk. But there's always three ways you can jump in and ask questions or provide your own perspective on topics too.
So you could jump in by raising your hand on zoom, you can put questions in the zoom chat, and you can put a little star next to it if you want me to read it instead, or else I'll just call on you to add some context and introduce yourself. And then lastly, we also have a slide of links where you can ask questions anonymously. And so our team will share that in the chat here in just a second too. Thank you all so much for being here today. And thank you so much, Josh, for joining. I feel like I wanted you to join us here for a long time. And I'm so excited to have you here today. Josh King is director of marketing data science and analytics at the knot worldwide. And Josh, maybe to get us going, it'd be great to have you introduce yourself and your role and and also something you like to do outside of work.
Sure, sounds good. And sorry that it's taken so long for me to actually land on the show. I feel like we've been playing a game of cat and mouse throughout that. No need to apologize. I think it was since like Domino's, maybe. I don't know. Yeah, it's been since Domino's, which was a couple of years ago. So yeah, I'm the director of marketing data science and analytics, or easier to say DSNA, or another way is saying marketing science. So I've been with the knot worldwide for about seven months, but it kind of really feels like it's going on seven years with just the pace that we move and the amount of cross departmental collaboration we have.
So we have a specific marketing focus in our group that's pretty new to the team and the team is pretty small right now because of that. So we have three analysts on my team, one data scientist. I'm also hiring for another data scientist. So if you're in the market, definitely feel free to reach out. So there's one position on my team focusing on marketing. We have another senior data scientist that's focused on operations based out of our Barcelona office. Mine can be remote anywhere in the US. We also have a product data science manager that we have open and also director of data science, basically my counterpart, but in operations that's also based out of Spain. So we have a lot of open roles right now. Feel free to reach out to chat about those.
And so my team at TKWW has an embedded nature and we partner really closely with everybody who rolls up to our CMO. And this is across the entirety of the world. So we don't just have like a US focus. We're a global company and we have a global purview within marketing data science, which is pretty nice. So I, prior to coming to TKWW, some people internally call it TKWW, which is even shorter and easier to say. Prior to coming here, I had about a year at a series E startup called Policy Genius, where I was basically doing the same function. That was a lot of fun, very fast moving, smaller company. And then prior to that, I had about four years of experience doing very similar thing at Domino's. So it's very different going from trying to sell pizza to trying to work in the business of love, which is pretty nice.
So one thing I like to do outside of work. So I love hanging out with my wife. She's also very close into the R community. She used to use SPSS for a lot of her analysis. She's an academic, she's a professor at Duke. And I got her to transition over to R now. She jokes around that she has a self-written package that she calls helper whenever she needs my help to help debug some of her code, which is pretty nice for her to have me as an asset like that. Also have two kids, a five-year-old son, almost 11-month-old daughter, and love to play hockey outside of work. So I play hockey one night a week in an adult league. Big fan of the Carolina Hurricanes, if anybody else here is in the Southeast. And yes, Louis, I think my kids will use R. My son already says that he wants to be a data scientist at five, which is pretty awesome.
I love it. That's awesome. Well, thank you. Thank you again, Josh, for joining us. I was putting on LinkedIn that I've been on a lot of the not websites in the past few weeks because I have eight weddings this year coming up. So I was just curious, could you share an example of a project that your team's working on or maybe how you use data at the not?
Data structure and marketing science at The Knot Worldwide
Yeah. So we have a pretty robust data infrastructure within the company. So I'd say like a year ago, our data function was set up in terms of pillars to have data competencies really be the pillars. So we had like a data science pillar and an analytics pillar and a data engineering pillar. And we had a new head of data startup with the company about 10 months ago. And he identified that there were some organizations within the company that didn't really have a lot of the strategic partnership within data. So he set it up to have more of a business competency focus in terms of pillars, which a lot of brands are doing these days as well. So now there's rather than based on data competencies, more focused on business. So we have myself as the head of our marketing data science and analytics function. We also have a head of product data science and analytics, head of operations that we're hiring for, and then also a data platform, which is basically all of our data engineers. So there's a pretty robust gamut of data problems that we're working on.
I think for marketing specifically, as I referenced, marketing didn't really have a significant amount of historical embeddedness in terms of data science. Maybe like there was an analyst that would support them, but not really in data science. So a lot of the models that we're working on building out right now are pretty foundational, at least from my perspective, to work with marketing. So we're currently focused on building out a media mix model, which I know a couple of weeks ago, there was a significant amount of discussion in here kind of about that and my interest in that model. And that model is really focused on using a lot of disparate data sources across all of our marketing channels to give marketing some good insight into an optimal spend amount across all of our channels to drive the most acquisitions as possible and the most efficient means that we can. So that's one aspect.
The data scientists that I'm hiring for right now is going to be focused a lot on driving some more engagement and solving our lifetime value problems. So I think some companies, especially if you consider back to dominoes, where if you want to drive an increase in lifetime value of your customers, then it's like, how do you get them to purchase once a week when they were once a month, or how do you get them to increase more of their pizzas within the orders that they're making? But within the not in our global brands, there's a pretty discreet amount of time that somebody's actually considered our customer because they come to our platform and they become a customer probably somewhere around the time they're getting engaged. And then by the time they get to their wedding, they're basically no longer really like revenue relevant for us anymore, unless they're doing something like thank you notes after the fact. So we have a pretty discreet amount of time for somebody to get engaged.
So how do we identify the value that we can get out of a couple as soon as we acquire them and getting them to engage along the appropriate manner? So it's gonna be pretty cross-functional working with not just within marketing to try to drive acquisition of high value customers, but also considering within product, how can we do some A-B testing to drive higher LTV couples into a more relevant experience and drive a lot of personalization?
Something I just figured out yesterday was the partnership with certain hotels on the websites. You can see who else that's going to that wedding is going to be staying at certain hotels. And I didn't know if that's another way of driving revenue too.
Yeah, so we do have, in terms of our revenue streams, there are a few different ones. And when a couple comes into our platform and they become a member, they can create a wedding website. And the website itself doesn't inherently drive any value for us. That's almost like a free offering that we offer. But then if you consider attaching your registry and then talking through the commissions that are built off of that one or something like that, let's start to get into some of them. And then a lot of it is our primary business model. And what we're focused on is on the overall wedding experience and helping couples do what it is they need to do to plan that huge moment in your life. So that could be what types of paper do you need to buy, which is invites, save the dates, thank you cards, et cetera, or building out a registry. Or the harder challenge is when you're planning a wedding, there's a lot of different vendors and venues you need to work with in order to just have your wedding go smoothly. So we are a dual-sided marketplace where couples can come into our platform and search for vendors that are local to their area based on their budget levels and what they're looking for. But then also we're working closely with the vendors themselves to have that aspect about, hey, here's how you can market within our platform and consider getting just in the eyes of the couples who are looking for you. And so that aspect of being a dual-sided marketplace from a data perspective allows for a lot of fun challenges because we're not just a B2C company, we're also a B2B company. So we have a lot of different challenges to try to approach that from both angles.
And so that aspect of being a dual-sided marketplace from a data perspective allows for a lot of fun challenges because we're not just a B2C company, we're also a B2B company.
That's really interesting. I'm seeing a few anonymous questions coming in and one was, do you notice a change in your work during wedding season, so spring to fall?
Yeah, so if wedding season is spring to fall, generally the height of our traffic as people who are coming through our website are generally going to be before that one. So that's probably when we'll start seeing a little bit of a lessening in terms of people that are coming into the site and actually engaging with it. So I think engagement season is generally considered around January or February. So we have a significant influx of membership during those times. And then most people are in the like eight to 14 month engagement period, we found. So as you start getting through, you can kind of consider the different journeys that people are going on based on not only just the time since they've registered, but also the time since they're getting married. And that kind of allows a good means of personalization in terms of sending people on the appropriate journey that's relevant for them based on both of those different event dates that are happening.
I see, Catherine, you put a question into the chat. Do you want to jump in? Yeah, I was curious as to what your analytics of, you know, speaking of wedding season, the lack of wedding season during the last several years due to COVID. And I said, you know, speaking of someone who had to postpone my own wedding twice during that time, what did that look like for you guys from like a data analytics and marketing perspective?
Yeah, that's a great question. So I wasn't here at the time. So I'm speaking somewhat secondhandedly about what I've heard through this and a lot. And it definitely changed our seasonal patterns in terms of when people were coming in, especially early on in the pandemic, when people weren't necessarily playing actively planning out weddings because they didn't know if they would be able to have a hosted wedding with like all their loved ones actually live in a room somewhere. So that made it a bit of a challenge in terms of couples coming into it. But we did see a lot of like pushed out planning to kind of to your point, like if you're getting delayed or postponed, then you may need to have a longer cycle that you're actually considering. One of the benefits is that weddings and like the generally called like the industry of love, like it's we're all about like celebrating these big moments in your life. It's fairly recession proof, which is a pretty good thing where there's always people that are getting married. And even if it's smaller, where they may not necessarily be spending as much, whether that is due to a recession or due to the pandemic, we do still see a significant amount of couples using our platform to help plan out the best event for them.
Marlene, I see your hands raised, too, if you want to jump in. Sure. Hi. Hi, Josh. Thanks for joining us today. Of course. You know, you you brought up kind of having a limited span or a limited time to engage with the client. Right. Has there ever been discussion about kind of expanding beyond the wedding season? Right. Because I think of it as OK, the main functionality is to stay in groups, engaged with the group that of your circle of friends and family or whatnot, just relevant people that you want to keep in the loop. Right. And there's other milestones that you'd want to keep them in the loop. You know, it could be children. It could be graduations. It could be huge promotions or moving out of the country. Right. So there are there thoughts about maybe they're not kind of expanding to something like the stork. Right. Where now you're announcing baby stuff. And I don't know, those were just thoughts that how to how to expand your client longevity. Right.
Yeah, that's a great question. So I think generally when I'm thinking about it and it's kind of just how the company is, our biggest market is in the wedding industry. So that's primary focus just because the global wedding industry on an annual basis is two hundred fifty billion dollars. So it's pretty significant market availability there. But we do. So within our brand, we do have many different companies across the world in 16 different countries, but not just focused on wedding, especially when you consider in the US, we also have sister companies that are part of our global domain for the bump in the bash. So the bump, to your point, is kind of focused on that next iteration that may be coming after you have your wedding, which is focused on being being a new parent. Like, what do you need to know and what are the different things you need to plan for there? And the bash is more focused on like just general celebration, like having having some
event planners or some party planners or some party party venues to celebrate any of those big moments that you may be considering. I think a lot of the focus is put on the former, which is more on the wedding. So just because that that is a very significant industry. But it is interesting as I've been recruiting for this position that I'm hiring for for a couple of weeks, I was talking to a candidate the other day and actually was talking to one yesterday who said he's going to be on the call today. So if you're here again. So I was talking to a candidate a couple of weeks ago, actually may have been the same candidate now that I'm thinking about it. And she brought up this aspect of, you know, after somebody gets married, maybe a year from now, just send them an email to say, hey, it's your anniversary. Congratulations. And like I think just like a general email like that, when somebody is our couple, it may not necessarily drive any revenue from us, but it drives a lot in terms of just customer mentality and having having us as on the forefront of their heads to go and talk to their friends and families about the services that we can provide and what we are.
Oh, Tripti, hello. She's here. I see somebody had asked earlier in Slido, is your team solely R focused or do you have more of a mixed code base? Yeah, we're very mixed right now. We have, I think a lot of our production code is living in R and a majority of the analytics that are happening are in R. We do have a fair amount that's also in Python. That's one of the reasons that I really like the reframing from RStudio deposit. It's kind of like it's just all inclusive. You can focus on anything you want. I think if you're focused on what language you're using, you're less focused on the business problem that you're trying to solve. So I definitely like the reframing there. But yeah, we do have a significant amount of code base built out in both of them.
I think you asked the question a little bit earlier. Do you want to jump in? And I can start reading the question too, but it was, so it was going back to the mixed media model, I think. So there could be various media mix objectives such as reach, frequency, or awareness. I'm not sure if I framed that the right way. I asked the question. So, Louis, feel free to jump in. Yeah, I guess just ask the question.
Yeah, I guess just asking, are those objectives your team thinks about? Is that kind of what you mean when you're doing media mix modeling? Or is it something different entirely?
It's definitely not different. And that's something we're thinking of. I think generally in terms of the MMM, this is a project that I kicked off with a data scientist that I hired back in November. So we're still pretty early on building out our initial proof concept for a single outcome variable for a single brand. I think as we continue to expand on this one and get a lot more iterative and bring out a bigger scale to a global model and also on different outcome variables, definitely awareness is one that's top of mind for us. So we have a significant amount of marketing spend right now that's focused on really capturing user intent, that they're already in the market and they're searching for something that's very relevant to our company. But considering a lot more, how do we start scaling to really drive awareness across those engaged or pre-engaged couples who will be engaged soon? So that we're already top of mind when they get to that stage rather than after the fact and find us after. It's a pretty significant benefit in the U.S. that The Knot has really strong brand awareness already, but it definitely could be better. WeddingWire, our other brand in the U.S., is still strong but not quite as strong as The Knot. So it's definitely one of the things we're always considering.
Fomenting empiricism
Something I know we were talking about a little bit earlier in preparing for this week, is you talked a little bit about how you drive and stir empirical thinking with your teams and marketing stakeholders. And I remember Googling like what that actually meant and now I don't remember. So can you tell me a little bit about that? Yeah, so I was working and so when I was working at Domino's, this is probably back in 2018, I went to this conference called the ARF or Advertising Research Foundation. And it's just a conference for people that work in the advertising or marketing space, whether that's advertisers themselves or brand marketers or data scientists to get together to kind of talk to best practices. And the keynote in that conference, the speaker the speaker said, I want you all to take away one thing today and that's to go back to your workplace and foment empiricism. And at the time, I didn't really understand what that meant. And then as I continued to kind of sit on that a little bit and let it marinate, I really thought about it. And because generally, foment has a very negative connotation, which is kind of like, I'm going to stir up and like cause some trouble. But if you consider fomenting empiricism, it's like within whatever workplace you're in or whatever setting you're in, just driving the thinking of having data and empirical evidence to support your decision making.
I think it really goes a long way for, as I'm talking to my team or talking to our business stakeholders to think through, we may have a gut feeling about something, but how can we think a little more analytically minded and try to challenge ourselves to see what the data would show us and try to drive a bit more value in terms of what's capable there, rather than kind of just taking things status quo.
I think it really goes a long way for, as I'm talking to my team or talking to our business stakeholders to think through, we may have a gut feeling about something, but how can we think a little more analytically minded and try to challenge ourselves to see what the data would show us and try to drive a bit more value in terms of what's capable there, rather than kind of just taking things status quo.
I'm guessing that's what you're referring to. Yes, that is. But I'm curious if there's like a time that comes to mind that your team did that or project for that helped the outcomes. Yeah, I think generally it's more of a mentality that I try to push with my team, especially as we get requests from stakeholders. So one thing is I try to push an embedded nature, especially within the analytics team. The data scientists team are more focused on like a project basis. The analysts are really embedded within the teams that we're working with. So this could range from brand media, social media, content, SEO, CRM, performance marketing. We work with a lot of different marketing departments and the analysts are basically aligned to some of these different aspects to really be embedded within those cultures and not just an outsider kind of looking in to try to get context after the fact when they have a question.
So as they're embedded, it allows for us to think as a team rather than say, hey, I'm a marketer. Hey, data team, I have this problem. Can you solve it for me? It's more, this is a problem that we have as a company. Now, how can we think together to better ideate this overall business problem? So the analysts really have a seat at the table to have a lot of that ideation to identify that we even have a business problem to begin with and then take that business problem and translate it now ourselves as the data team into a data problem and then build a data solution to solve that problem. But I think a lot of it is just coming back to having this mindset of making sure that we're keeping focused on the end goal that we're looking for and not necessarily say, hey, data team, can you go pull this data for us? The gut reaction is, sure, I can easily do that for you, but what's the intent of the question that you're trying to solve or the problem that you're trying to solve? And let us think a bit more holistically and as a partnership so we can think to try to drive a bit more solutioning and decisioning behind that because we, as data science and analytics, already have that mindset and know a bit more of what data can do. The stakeholders, even though they are very focused in having data drive their organization, they don't necessarily know what data can do. They know how they would want to analyze data, but it's not necessarily the right way.
So to kind of come back to your question, Rachel, about a specific instance, we received one question early on from one of our stakeholders when I came here, and it was, hey, can you go do a segmentation for us or can you build some clusters for us? And when he's like, I mean, yeah, we could, but what are the hypotheses that you have that you're trying to solve? What is the business problem you're trying to solve? And I didn't really have the context of the business at the time because I was only about a month in, but it really comes back to trying to identify what is it you're trying to solve out of this? If I deliver some segments for you, how are you going to use that to drive decisions? Are you going to have personalization? Is it going to change your audience strategy? Is it going to allow us to do some type of testing to make sure they're relevant? So I think just continuing to have those conversations with stakeholders allows it to be a bit more focused and less prescriptive from their part, and more so thinking a bit more generally about what can be done.
We were actually just talking about that a bit internally this week and thinking about the problem first before we start jumping into all these different solutions. But part of that was also like getting everybody to agree on what the problem is. Is that something you have to do across all teams? Yeah, it's definitely something we have to do. I think it's a bit easier when you take it back in terms of prioritization, because we kind of know as a company what our goal is over the next five years. And we all, as we're doing our own objective setting, we're making sure that they kind of align up to that.
So if we receive a question about, like, hey, can we do this analysis to see what's going to happen here? Whether that's like an analysis for which marketing campaign worked better or which specific channel on a specific campaign drove the most XYZ aspect within our brand. It kind of comes down to how is this going to help us align towards us reaching our goal? Because there's a significant amount of questions that we ourselves are asking ourselves, that our marketing team is asking, that our product team is asking, our operation team is asking. And there's a lot more questions than there are resources within the team. So it really comes down to having a very level head in terms of prioritization to make sure that we're focused on the right problems. And then if we can't align on there being a problem in itself, then I think that's one of those general things where it's, let's kind of think through what are some of our hypotheses and then continue to refine about what it is we're trying to get out of this. Sometimes that's just relevant for an exploration exercise to say, do we really have a problem or do we kind of just think there is?
Social media data and privacy constraints
Thank you. Anas, I see you have your hand raised. Do you want to jump in here? Sure. Josh, so I want to pick your brain on, I'm assuming that your company does use social media data to find prospective customers. And I'm just curious of, obviously, whatever you can share, if you use any other tools in that process, whether to feed into a model or, I don't know, that's a personal interest of mine, leveraging social media data and incorporating it into a model.
So, yeah, so I think there's a couple of different aspects there. One is generally within the newer privacy constraints, it's hard to do any type of user-level analysis where you necessarily know specifically, especially within social media, any actions that a user is taking, and then align, especially if you want to align that to something that's happening within our platform. So that's one aspect. As an aside, I'm somewhat generally for sharing my data, because as a data scientist and as a marketing-focused data scientist, I prefer to have ads that are 100% relevant to me so that I don't have to go searching for something else. I just, like, every single ad that I receive is something that I want or would need. So that's an aside.
The other aspect there is getting into, like, using social media data. So one specific aspect that you could utilize there, and this is, like, under the assumption that maybe you're working for a brand and you have some means of aligning on driving relevant people to whatever platform you may have, is by using signals that you can send back to things like Google or Facebook that allow them on their end to do some type of, like, look-alike audience building, where it's, like, hey, I've received these thousand people from your platform, and here's the click that they came into our site on. I don't necessarily know who they are, but I can say, like, this is the value that we see based on some predictive model we've built, and then send that value back to those social media channels so that they can, therefore, say, okay, I know how much you spent on this click. I know the value that your receiver met. Now I can help optimize the return on that spend that you're giving us to send more relevant customers back to you, and they can do that via look-alike audience modeling. Cool. Thank you. Yeah, sure.
There was actually a follow-up question that was asked, well, maybe not a follow-up. It was asked anonymously earlier, but have the recent privacy changes in browsers affected your MTA modeling if you do this? And I might need to know what MTA is and push your efforts forward and then. Yeah, so MTA stands for multi-touch attribution, which is a means of aligning all of the marketing touch points that a user could have before they make some action or they don't make an action. And so, yes, definitely privacy constraints are a significant blocker there. Even thinking back to, like, 2018 when we were doing this at Domino's, it was a significant blocker within the social media channels specifically that are very walled gardens. You can't align on an impression level basis who's seeing your ads. So it's definitely a challenge there.
One way that we're trying to work around that is by rather than building out a pure, what I would call MTA, which is generally using some type of Markov chain model that is impression level, that we're building out more of a click-based MTA, which aligns for a specific user. Again, this kind of comes down to our identity resolution and how well we can align any specific click to any specific individual for any of the clicks that they're making. What is the allocation that we may give to each of the marketing touch points that they have before they make some action? So that allows us to get a bit more perspective into some, what you may call, like, supporting marketing channels rather than, say, the last click before an action happens or the first click before an action happens. So you can kind of consider some different weighting scenarios there that may do it. The other aspect there was getting into the MMM. And this is kind of like, I'd say, over the last four or five years, there's been a bit of a resurgence in an MMM where prior to that, everybody was saying, oh, MMM's dead. It's such an archaic method that's being utilized to optimize marketing spend. And like, yeah, it was developed in the 60s, but it's still highly relevant, especially now as privacy concerns are coming out, especially within Europe where privacy laws are already well ahead of what they are in the U.S. So the benefit
to an MMM is that it doesn't require user-level data to do this. It's very looking at it from econometrics perspective and building out a model that can be very focused across the entirety and not just within digital channels, also within, like, TV and terrestrial radio and billboards to get a read into how all the marketing channels kind of work together.
Thank you. Martin, I see you have your hand raised. Nice to see you.
Nice to see you, Rachel. Hi, I'm Martin. I'm the CMO at union.ai. We do data and ML orchestrators. I used to work for our studio, so I know some people here.
So I think this is super interesting. I asked the last question, Josh, MTA versus a market chain-based MTAs and so on. But I see also the resurgence of MMM, but it has issues, right? I mean, if you have, like, time series that are short or sparsity of observations, it's not going to be good. So how do you overcome that?
Yeah, it's definitely a challenge you have to kind of work with.
One potential method, which, like, granted, it's not necessarily going to always be relevant, but if you're able to go further in granularity, then that can start to improve some of the sample size you have available in order to build out a model and allows you to do a bit more robust techniques. So right now, the model that we're building is, one, it's using open source framework from Facebook called Robin, which is pretty powerful, and it's built in R, and it's pretty out-of-the-box capable of doing pretty robust MMMs. Next steps beyond that are getting into more, when you think of, like, rigorously focused ones, getting into a hierarchical time series using Bayesian methods, which there's a package in R called rstan, which uses the stan framework to get into that Bayesian level. That's where we use the dominoes. And within that, as you continue to increase the levels of granularity, you can use the posterior distribution from one model to inform the priors, getting into another one, which allows you to kind of account for some of that sparsity you may have just by getting through there.
Another means is by, if you don't want to go further in granularity, you can actually do the opposite and start to aggregate some of those marketing channels that may be relevant from a consumer's perspective. So one of the considerations we made with the current model that we're focusing on at TKWW is by aggregating Pinterest and Instagram, which, like, they're completely different marketing channels, and they have, like, meta and Pinterest have no combination to each other, but from a consumer perspective, when they're considering wedding planning, they're both used as a means of getting visual inspiration for planning their wedding and seeing photos that are relevant to search for. So it makes sense to aggregate both of those together because of the sparsity that we had in some of those channels, which allows us to get a good insight to say, we may not be able to see what Pinterest is doing or what Instagram is doing in a silo, but we can say for visual inspiration, it has this return on our ad spend. And then as we get into further models later on, then you can get into that hierarchical analysis where you're able to then parse down what's actually happening within each of those channels specifically.
Collaborating across R and Python
Thanks. Caitlin, I see you had a question in the chat a bit earlier, too, if you want to jump in next. I can get started with the question, but it was, how do you collaborate between R and Python? Are these separate teams?
They're not separate teams. It kind of depends on what people are used to using. And that's definitely a struggle to consider when you want to do quality assurance on the code basis, but that's being written. If one of the data scientists is an expert in the problem that you're trying to solve or has some knowledge about it, but they know Python and I'm coding in R, then it definitely becomes a challenge. We're currently working on building out our overall data roadmap for our department on what is the appropriate means and scenario here for getting it and being able to have this cross-code capabilities.
I think one of the challenges there is, again, if you're focused on a specific language, you're not necessarily focused on the problem itself, which you can also relate to if you're focused on, hey, I just found this new novel technique. Let me see how I can pigeonhole it into the problem that I have. Then you're not really focused on the problem because that may not be as relevant. So I think one of the areas to do is just have some means of identifying who would be relevant to look at your code base, even if they're not necessarily on the same team. I've been at some companies in the past that were significant in size where there was basically an internal R user group and an internal Python user group, which allows for a lot of that language-specific collaboration and discussions.
Skills for the senior data science role
Thank you. One other anonymous question that came in is, what are the skills and tools that you're looking for in the senior data science role? Is MMM mandatory for this profile? And also, how could somebody reach out to you for this role? So I'm pretty active on LinkedIn, so definitely feel free to reach out to me there. And the role that I'm hiring for right now is I'm actually looking for, I'm not looking for MMM experience at all. So I have a data scientist that has a lot of fantastic MMM experience. I also have a lot of experience in that myself. So the person that we're looking for now is a bit more of a generalized data scientist, thinking through predictive modeling. If you have some experience in LTV or churn modeling, that's definitely in benefit, as our focus is on LTV. I think churn is a little less relevant because, just because of kind of like Hal was framing earlier about what our couples are going through. But if you have experience in that, it's definitely a benefit.
Creating the data and analytics roadmap
Thank you. Greg, I see you just put a question in chat. You want to jump in? And I always forget to say this, but if anyone wants to introduce themselves as well too, and add that in there. Hey, sure. Yeah, I can go ahead. So my name is Greg Schick. I'm a director of analytics for Charles Schwab, covering asset risk management.
And Josh, I just picked up on a key word you said there, is your data and analytics roadmap. And we kind of have a similar process. We manage two separate roadmaps. And I'm just wondering how logistically you manage that. Like what goes into creating that? How far out do you go? How far out do you try to plan? Does it cover all types of data work? So is it just more like a data science roadmap? Or do you include data engineering projects, BI, you know, ad hoc things? So if you can speak a little bit about that.
Sure. Yeah. So I'm actually in the midst right now, as in like the hour before this meeting, I was doing this on getting a bit more refinement on my roadmap, because we're having a data heads offsite next week in New York to kind of talk through holistically about each of our pillars, as well as the overall data work. So it's top of mind for me. So pretty, pretty relevant timing there.
So one of the things I like to come back to is the CTO of my last company, he had this really good framework for kind of thinking through this, and it's to break it out into your mission, your vision, your maturity model, and your strategy. And that kind of allows a bit. And so if you want to look it up, he has a great article on Medium. His name is Dave Kaplan. And he has some good articles on that. But if you search for those keywords and Dave Kaplan, there's a really great posting on it. This kind of gets into a good framework for building out and identifying like, what are the different aspects of a strategy that you should consider, and having a clear alignment for what that will look like.
So one of the things I like to come back to is the CTO of my last company, he had this really good framework for kind of thinking through this, and it's to break it out into your mission, your vision, your maturity model, and your strategy.
And generally, a mission is like a single sentence for why your team or your department or your company exists. And your vision is, what does it mean to succeed within that? So like, if I'm thinking through marketing DSNA, I'm thinking through like, what does it mean to have a best in class marketing data science organization within like across relevant brands? And then your maturity model is, where are you along that map from like very basic or early foundational building into like, I'm best in class. And then your strategy is how you get from where you are now to where you want to be in your vision. So this really aligns a good framework for what it is that you're building out.
And to the other part of your question, like, does it cover all data work? So I have data scientists and analytics that report into me, but there's also some of a dotted line from all of our data engineers that focus on marketing problems. So that can include BI, that can include data operations, data warehouse, tracking architecture. And as we're thinking through a bit more holistically in terms of all of our data functions that are supporting marketing, I identify like, what do we need to drive decision-making for our marketing organization? And not only like, how does data science and analytics fit into that, but like, what is the infrastructure we need from a data engineering perspective that would allow us and enable us to be able to do this work? So that it's not just focused on that one, but then yes, it's not necessarily my vision or my strategy that would get us there, but working really closely with our head of data platform so that my vision and his vision are really aligned together and thinking through a bit more holistically as we're building that out.
And I think the other question was like, how long is it? So my current focus is on building out a one-year, a three-year, and a five-year roadmap to really have some like intermediate stages along the way that we're getting towards.
Gotcha. That's impressive. We're working on the one to three-year. We haven't got to the five-year yet, but that's pretty cool. I think like it's a bit of a struggle because especially within data science, like it's so rapidly evolving that it's hard to think through like what's even going to be the atmosphere and the environment in five years from now.
Yeah, absolutely. The other struggle we seem to see is the need for some of the foundational data engineering work to come first, to enable some of those capabilities, right? We've got a million requests to add chat GPT to something and we're like, well, we need to build some data infrastructure first. So it's not as easy sell, I think, to do that foundational work first.
Cool. I appreciate that. Yeah, it's definitely a bit challenging in terms of orchestration to make sure that all those pieces kind of fit together. So as I build out my roadmap, there are some things that can be done now based on the infrastructure we have and some things that like, yeah, it'd be great to do it now, but really this would need to be on the immediate roadmap for data engineering to build out what we need in order for us to do that even a year from now.
Experimentation and randomized control experiments
Absolutely. Thanks. Thank you. Curtis, I know you had a question a bit earlier too, if you want to jump in next. Yeah, I just had kind of a methodology question. I was kind of curious what kind of randomized control experiments, like control experiments you guys are running, if you could talk about that and particularly how you integrate that into your analytics pipelines, marketing strategy. You talked a lot about fomenting empiricism and I've heard a lot of things of, you know, kind of prediction models, but I was just kind of curious about the experiments you might run, that sort of thing. I didn't hear you talk about that yet.
Yeah, so within marketing, the experimentation is really being done somewhat siloed within the marketing channels currently and that like Facebook or Google, they have their own abilities to do a lot of testing. We can do some analysis on the back end to do like more down-funnel experiments. Excuse me.
North Carolina pollen. So we're getting into now, this is one of the things on like the next couple quarter roadmap, is to build out more of a geo-based experimentation framework, which allows, like it's not randomized, but allows us to have a good amount of insight for a specific geography or a sub-geography and then have a relevant control set to compare against. I say a lot of those RCTs happening within our product side and getting through people who are actually within our platform to try to drive a lot of that experimentation via A-B testing, and it's not even necessarily A-B testing, like multivariate testing as well, to really drive a lot of that like personalization, what works well as we're making changes to the website.
Building cross-departmental relationships
Thank you. I know a bit earlier you talked about how you work across, like cross-departmental partnerships, and a lot of times we'll talk about like how you build relationships across the company in this data science hangout format, and I'm just curious what that looks like at The Knot.
Yeah, so I try to make a specific intent to, like especially since as I was going through my first six months, really building out a lot of the relationships with the people that I'd be working with or like partnering generally close with, and a lot of that is just like communicating on Slack to say like, hey, what's going on, or having regular touch bases and one-on-ones. There's also, if you consider within whatever organization you're in, if there's ever like one-on-one organization you're in, if there's ever like lunch and learns or any aspect to hear from others' perspective, go to those and ask questions. I think like that's one way just to kind of get out there and start to build some of those relationships is by just being there and hearing other people's perspectives, and if they ask, do you have any questions, like try to have some type of question that you're able to ask because I think that's just a good way to try to get your voice heard and start to build a lot of that relationship.
Omni-channel data and problem solving
Absolutely. Thank you. Jonathan, I see you asked the question, and there's a star, so I'll read it, but the question was, does your team use omni-channel data for driving revenue and solving problems within your teams? If so, then what is a typical issue faced in your area and your approach to problem solving?
Yeah, so we definitely are focused on omni-channel experiences and try to make sure that any marketing experience that somebody's getting is relevant across any marketing touchpoint that they receive, so a primary focus that we're utilizing to do that is via the CDP or customer data platform, which allows us to orchestrate journeys to make sure that we're sending content relevant to the experience that somebody's had within our platform and what they're interested in based on like what keywords they're searching for or something like that to make sure that if they're getting an ad on Google and an ad on Facebook that the copy is relevant to them and it's relevant to whatever it is that they're doing, so that CDP has really been a significant aspect allowing us to get into a lot
more personalization based on specific events that are actually happening within our website.
And just to double check, CDP, that's customer data profile? Customer data platform. So it's like a singular location to store profiling variables about customers that you know as well as all the actions that are actually happening within your website, and you can build filters and do custom SQL traits and really get a good perspective about how people are interacting and then build custom journeys based on how those interactions are actually happening. Cool, thank you. There is an anonymous question, I just had to actually Google this real quick, but what has your company done with generative AI, if anything?
Generative AI and personalization
So nothing yet. That is something that we are considering getting into. This is an idea that I had early on that I kind of pitched to our product side. If we have, say, a woman shopping for wedding dresses and she has a photo of herself, can we have within our product the ability to allow her to completely visualize herself in 10 different wedding dresses? Or if somebody has a specific color profile that they want to have and see how flowers would be, and they also have something else, allow it to do some of this color picking across different palettes to allow a very personalized wedding planning experience. So it's definitely something that's on our radar as we're trying to keep up with a lot of the ever-evolving trends, but nothing that we're currently actively using. It's making me think of, like, when you buy something on Wayfair and you can, like, take a picture of your, like, house and see what the furniture looks like. Exactly.
Diversity, inclusion, and data
Let me just double check here, I haven't missed any of the anonymous questions, but I think Catherine, you had another question in the chat, too, if you want to jump in. Yeah, and it may be a little charged for this format, so feel free to cage however you'd like. But just curious with how many aspects of weddings are now legally and politically, socially complex, what sort of, like, are your guys' marketing and engagement teams doing maybe in, like, political action or other sorts of social community activities using some of the data that you collect? Yeah, that's a really great question. So I don't necessarily know about utilizing any of the data that we collect, but we definitely, as a company and our CEO himself, have come out in our support of gay marriage and making sure that everybody's feeling inclusive. So, like, as a couple years ago, as there's a lot of that discussion along the political atmosphere, like, there's a lot of statements coming from our company.
So we, I don't know that we're necessarily utilizing any data. We do have some diversity filters that are able to be utilized within this, within our platform to say, like, hey, if I'm an LGBTQ couple and I want to make sure that I'm supporting the LGBTQ vendor, then, like, I can filter on that one, just to kind of, like, keep, like, for like, and look for those focus areas. But we're not necessarily doing anything in terms of an analysis or data science perspective to share anything out there. I think it could be a pretty good idea for some type of, like, data journalism within some of our editorial content to look at the different trends that are happening there, though. Yeah, I actually, you know, am not a member of the LGBT community, but I was getting married in Texas and it was important to me to find an officiant who didn't say, you know, no LGBT weddings on their website. And that kind of thing can be really important and just, like, a really powerful, simple filter that people want. Or I guess some people want to filter the other way as well, so.
Career advice
Thank you. A question that I always like to ask Josh is, is there a piece of career advice that you've ever received or maybe that you've given that stands out to you? Oh, that's a good one. Well, I kind of actually already used one of them, which was if you're in a meeting and you're asked, if you have any questions, then ask a question. But I also, I feel like I can't repurpose that one, but maybe I could punt and say that I would. But I think there is something that's somewhat relevant to that, even though it's not necessarily a piece of advice, but like, just a general mannerism. If you're giving a presentation and you get to that and you say, do you have any questions? It's much easier to get some engagement from others if you reframe that and say, what questions do you have? I think, like, generally, and I've seen this also from experience, granted I didn't set it up as an A-B test, but you get a lot more relevance coming out of it and engagement if you say, what questions do you have? And people are more likely to do that. Also, if you ask somebody, what questions do you have, or do you have any questions, count to 17 in your head. It's not like one, two, three, four. It's like one, two, three, four, five, six, seven, getting to 17. And that pause is sufficient enough to allow somebody to kind of wrap their thoughts together and build out something so that they can actually have something ready for you. A lot of times somebody will say, do you have any questions? Okay, good, and then see you later. But actually having that specific pause in there and actually doing that mental counting in your head allows for a decent amount of time to actually get there.
And that pause is sufficient enough to allow somebody to kind of wrap their thoughts together and build out something so that they can actually have something ready for you.
The thought of waiting for 17 seconds there just gave me a little bit of anxiety, but are there any questions? Yeah, I've called that out specifically to my team so that if they feel a very long pause for me, that's exactly what I'm doing, because I heard this piece of advice at one point. And so it's not that I'm just like sitting there waiting. I'm waiting indefinitely. It's just I have this specific pause. Yeah, and with Zoom, it's not like, oh, maybe he's having audio difficulty. Right, exactly.
So actually, are there any questions that I have missed in the Zoom chat here? Feel free to just raise your hand if there's something that you want to bring to the top or something we missed. I actually did just see one come in on Slido right now that was, what important roles or opportunities have you taken throughout your career that has helped shape who you are today? Oh, that's a really great question.
Career-shaping experiences
So I think there was one. So I used to work at Domino's, and I was promoted to director. And then I think like six months later, so my wife was a professor at University of Michigan at that point, but we're both from the Carolinas. And Duke University reached out to her and ended up offering her position. And so we were moving back to the Carolinas to be closer to family. And then I wasn't really finding the right job in an industry that I felt really strong about. But as I was just kind of talking to people that I was meeting through my wife's academic relationships, I met these. At Duke, they had a new program called Messer, an interdisciplinary data science. And within that one, I really loved kind of like the framing that they were doing. And they're looking for somebody who had a lot of industry experience to come in and be faculty to not just like talk about the theoretics behind the machine learning applications that are happening, which like at Duke, there's a lot of amazing professors that can help drive a lot of that. But they're looking for somebody that had a lot of that like software skill aspect that you can find with a strong data scientist to work through project management and build relationships and ask the right questions and like effective storytelling, et cetera.
So because I wasn't finding the right job in industry, I wanted to go with that one. And sadly, my wife was like, oh, you're going to hate academia. It's not like working in the industry. I'm like, no, no, it's going to be great. And then I got in there and like it was very changing for me. I saw a lot of different perspectives and was able to get a lot of a different like project experience as I was doing that. I did find pretty quickly that like academia is just not for me. But like I still feel like it was something that really changed me for the better as I was working with a lot of these grad students that were really early on in building their career and seeing a lot of these different projects that were being worked on.
Thank you. I do see one other question that came in in the chat from Jonathan. And Jonathan said, for someone that's new to the field of data science and has experience using R and Python with banking, what are helpful resources, books, or communities you recommend that could support with that transition to data science?
Yes, my master's was actually in computational finance. So it's something that like I have had a bit of focus on it, but a lot of it wasn't like the actual mathematics side, like building out some pretty intense math theorems for them and stuff like that. But like one of the books that I go to even still now is, and I can follow up after and send this to you, but it's like a basic econometric or advanced econometrics using R book. I think that's one that I go to a lot. And econometrics is very heavily used within finance. That's getting into a lot of the different types of time series analyses. I wouldn't have as much recommendations if you're trying to get into like algorithmic trading. I had a class in that one, but it wasn't highly interesting to me. It's an amazingly interesting subject, but not one of those ones that I'm an expert in.
Thank you very much. I was just asking in the chat if anybody can find that book. Feel free to put it there in the chat. I was trying to Google at the same time. But hey, Trevor, I see your hand. Advanced microeconometrics or something like that. And it was all in R. Awesome. Hey, Trevor, I see your hand raised. Do you want to jump in here next? Yeah. Hi, Rachel. Hi, everyone. Hi, Josh.
The impact of AI on data science roles
So my question, I'd made a statement before in the chat, but I'm turning it into a question, right? So last week, I was asking about the impact of AI on data science, right? So in that vein. Where do you think, Josh, and it's just your opinion, given the advance of AI, and we clearly see AI is making great strides in terms of writing codes, right? Writing code. Do you think that most of our time will be spent more on qualitative things, such as defining, assisting stakeholders in defining the problems, as opposed to thinking about the latest way to code, or thinking about the latest algorithm, or anything like that? Do you see a shift, or a balance, or a shift in terms of where we spend our time as data scientists?
Yeah, that's a really engaging question. I think there's a lot of hot takes on that. I think for the immediate future, if nothing else, it's going to allow data scientists to probably code a bit more efficiently. I think a significant challenge, and I haven't asked ChatGPT to go in and build something, like write a piece of code for me. I think my wife has done this, which maybe alleviates my requirement to utilize helper for her. Along those lines, I think it's going to allow data scientists to be more efficient as they're coding. I'd say it's probably five to seven years until I can actually do anything substantially relevant to consider data scientists are only working on framing problems in a way that AI can interpret them, and be able to be embedded on top of your snowflake warehouse, so that you can just ask a question, then it'll go through and build it a model. I think we're a long way away from that. Okay, thanks. It's amazing to think about, but from my own career perspective, I hope we don't get there anytime soon.
Omni-channel marketing examples
So, random question that just came to mind, but we talked about omni-channel marketing, and different marketing analysis that you do, but I'm curious if there's a company that you think is doing a really good job at this, that is marketing to you individually.
Yeah, so I received an email a couple days ago from Verizon, and I just like, generally, I pay no attention to them, and so I utilize Verizon as my service, but I clicked on the link, and then immediately after, I saw when I was watching the Hurricanes game last night, that there were several ads that were all focused on Verizon, and then I received another one today, as I was on my computer, so their omni-channel experience is top-notch, and able to be connected across all these different devices, so that's just one that's top of mind, as I've been noticing over the past couple days.
Well, thank you. I know we're getting to the end of the time here, but I want to make sure that I share the roles that you have open, and just put those into the chat here. I found the one, so I think I have the one for senior data scientist, which is here on LinkedIn. Yeah, so we should, but if you have a general career stage, I should probably go to that. I think I may actually have that up somewhere, because I was looking at them earlier to make sure I had all of them right. I'm not seeing it. I have too many tabs open right now, but I can put it with the recording too, and I'm sure we can navigate to the careers page. Yeah, I'll follow up with you after that one, but if you go to the notww.com, that's our careers page, and then search for data, we have two open senior data scientist positions. One is based out of Spain, one is remote anywhere in the U.S., which is the one on my team. We have a product data science manager position that's remote, open anywhere in the U.S., and then also a data science director position that is based out of Barcelona. Perfect. Well, thank you so much, Josh, for joining us today and sharing your experience. Thank you all for all the great questions.