image: thumbnail.jpg
Data Science Hangout | Moody Hadi at S&P Global | Unlocking Business Value with Data Science
The Data Science Hangout is a weekly, free-to-join open conversation for current and aspiring data science leaders.
An accomplished leader in the space will join us each week and answer whatever questions the audience may have.
We were recently joined by Moody Hadi, Manager of New Product Development and Financial Engineering at S&P Intelligence
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.rstudio.com
LinkedIn: https://www.linkedin.com/company/rstu...
Twitter: https://twitter.com/rstudio
Transcript#
This transcript was generated automatically and may contain errors.
So S&P Global really has basically five business lines now. There's the Rating Agency, there's the S&P 500 and the Dow Jones Index Business. There's PLATS, which is an energy focused basically side of the company, like a lot of research on fracking and oils. And Market Intelligence, which is basically kind of like a data analytics vendor. And then the fifth one is relatively new, it's called Sustainable One. It's completely an organization focused around ESG data and finance. So in that area, I run the new product development. So I'm technically under the product management group. So I leverage a lot of the technology folks in data science, a lot of folks in what we call content, which is like automation teams that do more RPA work. And of course, my own team, which is technically generally more quantitative, but in a more user oriented fashion, I should say.
So when we say when I'm talking about new products for S&P, it's typically products that take about six months to a year to go to market. So they're not like feature enhancements or things like that, that are like much more transformational. So for example, we've launched a financial statement extraction system that basically take a document that's completely sitting on your desktop, and then you upload it and it becomes part of our platform. Our sentiment analytics is for Chinese companies in the native language, simplified Chinese. So things like that, that are not typically what you might necessarily associate with the S&P sort of product offerings. So yeah, we use a lot of, you know, we have R in production. We also use, of course, a lot of the NLP tools in Python. So we kind of cross all the usual buzzwords in the data science world. So hopefully that gives you an idea. Yeah.
That's great. Thank you, Rudy. I see Rahul, you asked if there was a Slack channel. I was just mentioning to our internal team, just to share the link to the Slido. No, we don't have one yet. Maybe someday. But also just wanted to point out, because I didn't say in the beginning, and I realized the recording was paused, but we are recording this session as well. So we can share it up to YouTube after. So if you want to share it with anyone who missed it as well. And that link that they put into the chat is the Slido link. So if you maybe didn't want your question to be in a recording or anything, you could ask anonymous questions as well.
But love to have people jump in. So if there's certain questions that you'd want to ask to Rudy as well, feel free to put those in the chat. But Rudy, I'd love to maybe get started with asking you about some of the things that you're really excited about in data science right now.
What's exciting in data science
Yeah, sure. I mean, yeah, like I started kind of in the quantitative side for about like in about two decades now. So the exciting part I see in the data science community is the ability for folks to get a lot closer to the domains that they're working in. And I think a lot of the work that RStudio has been doing, especially on like making building applications easy in R actually helps sort of get that point across. Like in my world, obviously, in the quantitative world, you have a lot of folks who kind of just sit behind, look at numbers. They don't know what those numbers mean and how their client will use them. So I've seen over the last at least five years, there's a lot more transformation there where the sort of the upcoming data scientists actually try to think beyond the numbers and how the client will use them.
And that's sort of, I think I've been in finance for a while. So in finance, you get a lot of folks kind of like that. But yeah, that's exciting. That's nice to like talk to higher people who actually think already from out of grad school like that. That's great, right?
I think more and more technical basis, like ability to like visualize information very easily with like very few lines of code. Like a lot of what I do on the prototyping side, we use Connect, we use Dash. But the net result is there's very few lines of code that you basically have to write in order to get something that looks like a legacy, like a JavaScript based platform, but kind of actually more cutting edge, I would say. And that's important, especially when you're trying to kind of get to the last mile, because typically in bigger corporations, to build sort of those type of applications, they go on this Chrome agile release cycle takes quite a few months to get to that point just to show a client something interactive.
And then the other thing that excites me, third thing is I think more on the machine learning side that neural networks, obviously very, very big black boxes, but they can mine a lot of data and take advantage versus say structural models, right? The explainability side of it has become a lot more, I would say a lot easier to actually do. Like we use Lime a lot for model agnostic explanations, like simple things like that kind of go a long way to sort of like, here's a nice time series, but why? Like that question gets kind of answered nicely, and then ties back to the visualization, because we kind of build things that try to simplify that answer without having to, without having the customer to reach out to support or to other sort of sides, or, or worst case is having the client have to jump on a call. You kind of want things to be more on demand self service. And I think that that side of it, of course, there's a lot more still sort of infancy, but like in the middle of mid level, I think it will get easier and easier as time goes by, at least I hope so.
Career journey and industry changes
Yeah. When you think of that whole process and transformation, the team's taken, how did you get to where you are now? And like, what did that look like in the beginning? I mean, I started like you mean, me personally, what my how my career kind of progressing? Well, you were saying like, over the past five years, some of the changes and things that have, oh, yeah, I meant more like, yeah, the changes. Yeah, I think it was the reason for the changes, I think has the, you know, obviously, a lot of cloud computing costs have gone down, like back in the day, you'd have to have used to host your own on prem server, that costs a lot of money to actually do this. So AWS has lowered that cost a lot. So anybody can now out of an apartment, kind of launch something that needs multi GPU, multi CPU things at relatively minimal cost.
So that lowered that then the applications, I think that the languages that are available, like R and Python, I think, Python tends to be a bit more on the software engineering side, but R especially tends to be the other way more than stats slash animal side. And with RStudio, you guys have done a lot of really neat packages that made it even easier to write code. So I think people have combined that computational piece for the lower like easier to understand, or semi functional language. And they're able to kind of get beyond just the numbers and actually deal with the data itself for the analysis. I think that those two things have kind of low made it easier for for folks to sort of at least have an idea about about what they're looking at. It's not I mean, it's not a general rule, like you still have a lot of people that don't. But I think that's that's part of it. But yeah, I mean, it's a very bright future, I think in that area. So to answer your questions.
Sharing applications and communicating results
Yeah, no, no, that does. I don't see any questions coming in from anyone else yet. So I'm going to keep asking questions if that's okay. But I know you spoke a little bit about some of the visualizations that you're making on the team, whether that's in Python or in R. Who do you typically share applications out with? And what does that communication process look like? Yeah, yeah, that makes a lot of like, so for me, it's two sides, right. So one, like internal where and by internal folks, I mean, like our salespeople demoing something to a prospect. So we just, it's a new product, we're still kind of assessing the market appetite, the addressable market, basically.
So we build it in house, you know, comes over the VPN, it's going to simple URL and connect, typically. And they would go out and talk to clients and showcase basically the visualizations, well, not really visualization, but like the workflow. It's really workflow based tools, right. And then depending on the feedback, that's when we kind of figured out what the best channel to put it on, whether it's a feed, or it's a desktop offering, right. That, of course, comes with a PowerPoint deck, right. The other one, which is more interesting, at least to me, is that the external piece. So we have done, we do, we have something called the incubator at marketintelligence.com. That's an external facing, basically, pre-prod environment, if you will, it's, we can sign up clients and prospects on it under some, you know, SLA type of deal. And that is completely, whoever you sign up, they go in and do their business using that application. And that's also serviced by, in some cases, Connect, depending on how many users we have concurrently, and other cases by like Dash and Plotly. But so that's, that's kind of how we kind of do it. But again, all of those have some lifetime to them. Eventually, they, within six months to a year, they go back into the platform offering, which is all in JavaScript, basically.
And, and thinking about those, I guess, as follow up to that question, I see there's a question on Slido. Do your clients set standards for which tools and approaches they need you to use? Or do they just care about end results, and you're free to choose?
It depends. They have standards more, not, I mean, not the workflow, necessarily, workflow needs to be, quote, unquote, intuitive, right? So that unfortunately goes through a lot of like iterations, right? Until it becomes intuitive. But like, yeah, depending on a product, so like, in like credit analytics, for example, you can't just basically arbitrarily decide to build a neural network to do, you know, implied ratinging. Because that comes with, although it's not a ratings product, it is not a ratings, S&P ratings product, it's lowercase letters that, that goes through a lot of audit, a lot of stability criteria. So that you can't just arbitrarily decide and change the model, like, but yeah, so they're like NLP things that I don't think they care, like we have the text and the transcripts filings that we have that does basically takes an earnings call, converts it into, you know, machinery double text, like from voice, and then runs a bag of words, regularized bag of words against it. Now, that could have also been done with a neural network, just regularized bag of words was easier to explain. The China sentiment analytics, on the other hand, is completely neural network based, right? And with like, model explanation on top of it. So that we have kind of freedom for the for sort of the unregulated sides of the business, right? The things that are regulated, we don't have much on the on the visualization or the workflow side. That's where a lot of the opinions kind of come into play, right? That's like the output means something, but it's hard for me to say upload my portfolio and get back my output in like two minutes, you know, whatever it is, right? Or, you know, I don't really understand why this widget kind of looks like Excel, but it's not.
You know, so those are the things that typically take a lot of time to sort of convert into something easier for the client to use. But that sometimes goes gets out of scope of the quote unquote MVP or a prototype piece, like, you know, some of it, you just don't have enough, like it becomes too much customized for Connect or Python to do right. So, you know, but, you know.
Do they typically have some sort, like, do you have a meeting with them about maybe what the end product would look like for them? And?
Yeah, so like, typically, you start with like, you start with your own guests, like you, because we know our client base well enough. And we, we obviously they use our desktop platform. So we take a first stab at it. We try to avoid like back in a few years ago, we were building the stuff in Balsamiq and things like that. It's less meaningful to the client. So we tend to get to the prototype, something at least functional, or fully functional, like real data, real numbers, real model, before we show it even to the first friendly prospects.
And then we iterate with them based on their basic feedback. And then eventually, we reach some kind of, like, you know, I like to say, everybody's equally unhappy, you know, that everybody's, they can get their business done, it may not be perfect, but it's, it's there enough, right. And that's when we kind of push it out so that they can use it and go about it and then collect the new feedback for the product side of it, like for the actual legacy platform to basically improve.
I like to say, everybody's equally unhappy, you know, that everybody's, they can get their business done, it may not be perfect, but it's, it's there enough, right.
But yeah, it's highly like, it's a lot of, a lot of folks, a lot of talking directly to the client.
Greatest challenges
Shifting gears a little bit, I see another question that came in anonymously on Slido is, what is your greatest challenge right now? And how are you tackling this?
I mean, right now, honestly, I would say the biggest one is hiring, like, we need to hire more sort of qualified data scientists. Like, we obviously have a team, we have several teams. And I think that's one of my biggest, like, items to address for basically 2022. On the more technical side, I think explainability. Like, this is a problem, especially, like, as we start dealing with this big data sort of purge, or like this ability to mine big data, for lack of
better term, like, it's always nice to see a time series of some input, but the reality is the end user typically won't take that as gospel, right? So, trying to kind of explainability, like, so, trying to kind of explain why, and especially with the subjectivity part of it, like, you know, we have folks who have an opinion about a particular market and other folks who have a completely contrary opinion about the market, yet you're trying to set something that works for both. I think trying to be very transparent and sort of unbiased, this is important. And that's where the model explanation comes into play. At least so long as both sides understand where you're coming from and it's defendable, it's self-consistent, then you don't run into this sort of problem.
But that is sort of evolving. Like, in finance and, like, financial engineering, I mean, you can tell, like, there's some form of absoluteness to the values that you produce, right? Like, a lot of structural models, a lot of implied implicit assumptions. So, there's, you know, somebody wins, somebody loses, right? In some of this sort of weak predictor area, there isn't that. The only thing that you can kind of tell is that I've looked at something, and this is how it works, and this is why I put it at that level. I think that piece, people forget about, like, especially sentiment analysis. They think there's some absolute measure, but really, it really depends. I think the question, there's no such thing as a better model. It's more like, what's the context, and is it self-consistent, right? Those are the two things that you want to keep in mind.
There's no such thing as a better model. It's more like, what's the context, and is it self-consistent, right? Those are the two things that you want to keep in mind.
The hiring mismatch in data science
I think it's interesting how, I mean, the first one you mentioned was around hiring, but I see so many different jobs out there on LinkedIn, and I try to share as many as I can, and then I also see so many people who are looking for jobs right now as well, and it feels like there's some, like, mismatch happening there, and I know, Ian, you you can speak to this a little bit as well, but I'm just curious to, like, chat a bit more about that. Like, what is going on there, and why is it such a challenge?
You know, I think it depends on what you're looking for, right? Like, that's part of it. I mean, on both sides, the applicant and the, basically, the company that's hiring, the hiring company. I think some of the applicants, I mean, SMP is a pretty big company. They just don't want to work for every company, you know, so that you kind of run into that problem. On the other side, like, the teams that I deal with, they have to be, like, basically, they have, I think they need to know both R and Python well enough to at least articulate, at least to use them for, basically, exploratory data analysis, and R and Python, a little bit of, basically, distributed programming, and that's very rare. I think, generally, people gravitate out of grad school, they tend to gravitate to one or the other, and it's, like, hard to kind of switch that mentality. I think that's, I think those, at least in my experience, I think those are the two major roadblocks to that sort of mismatch.
Frank, I see that you're unmuted. I'm going to take that as a hand raise. I totally agree with everything you just said, right? That division, that R-Python split. No, you use, you know both, you can talk to both, because you'll need different tools for different projects, and then, ultimately, I'm just wondering who on your team is thriving, right? What makes a team, because you're talking about all these different projects that you're doing in sort of different ways. It feels like your work isn't as streamlined as, probably, we all want it to be at this point, right? It's evolving, it's developing. Can you describe some of the folks in your team that are just phenomenal and doing really well in that matter?
Yeah, no, no, I hear you. Yeah, I think a lot of it is, like, for, again, like, the ones that are thriving, I mean, they come in with a pretty strong technical background, both in math and programming, right? So, the ones that kind of move up faster are the ones that understand what you're trying to do with it more, because, as you can imagine, we're, you know, in some days you're working on some, you know, sustainable development goal thing, and another day you're working on some sentiment analysis, and another day you're working structural model. So, the ones that actually get what you're trying to do, we put them in front of clients, and they thrive by becoming a bit, again, a lot more product management. They're able to explain it as well as a product manager, and, but they also, of course, have that technical background. So, those are the ones that move up.
Then the second side of it is the management piece. So, because we are, my team is relatively small, we rely on some of the IP folks, we rely on the content, like the, what we call contents, like data acquisition, and those are also technical folks. So, being able to sort of manage those sort of dotted line teams and still meet your deadlines without slippage, those two things kind of move somebody up on the scale. On the exploratory data analysis side, like, we do have some more, like, I would say creative topics that come in where we don't know if there's a product, but we bought the data anyway, you know. So, we have folks who kind of thrive on trying, like, multiple different types of models to see if they can actually mine it for something useful. But that in conjunction with basically the understanding of the use case is sort of important.
So, those are the other side of it. Do you ever sacrifice, so when you say they're more technical, my mind immediately goes to PhD in math or PhD in finance or something, or like PhD in physics, right, something like that. Do you ever sacrifice on that end of the, let's say, academic spectrum, right, if you can find people who know the technical document in R-Python, right, they can do the analysis there. And they're also, like, very keyed into the business, so they can do that communication piece very fluidly.
Yeah, yeah. I typically, I would do grad school above. I don't really specifically look for PhDs, not in a bad way, it's just, you know, it depends what they want to do. And then finance too, they, in finance, the banks typically have a lot more, what's it called, like, more structured paths for the PhD side of it, quant development side, and some folks just want to do that. Again, we're like, we're more like a fintech than a fin or tech, you know. So, like, there's not necessarily as rigid a path as the banks, right, for sort of that track. But yeah, I mean, ability to learn quickly, I mean, you know, the languages change, you know, the business changes, being able to adapt is more important to me than basically, you know, I mean, obviously, there's a bare minimum, right, but between that and just, like, pure technical, I'd rather have someone who can adapt, to be honest.
Non-technical skills and hiring process
As a follow-up to that, and some of the non-technical skills, I see Millie asked a question about that. Millie, if you feel comfortable jumping in and introducing yourself, would you want to ask that question? I can also ask too. But it says, what are the non-technical skills and values that you have when looking for data scientists?
Non-technical, yeah, like, so a lot of it is more like presentation skills, right, being a little, you know, articulate, like succinct, right, some writing skills, like we do a lot of thought leadership and things like that. So, just trying to sort of, again, do like, used to be called techno-marketing, but white papers, you know, some form of white paper type things. So, there's that level that you want to have.
I mean, honestly, not all of it comes, like, out of school, you're not going to have all that. So, typically, we end up bringing the junior person on calls, mute, just shadowing to see how that articulation, and lately, I've been putting folks more on internal calls with, like, kind of pure business side folks, business side colleagues, and kind of, where basically, if they make a mistake or say something kind of funny, but it's not, you know, there's no real problem. Eventually, they kind of graduate up to, like, a client call, right?
Yeah, again, I don't expect people, like, at least, I don't really, if you were working other firms, I do kind of expect some level there, but if you're coming out of, like, grad school, I wouldn't necessarily, I'm not super worried about it so long as you can kind of adapt. A lot of listening, actually, is more.
You rarely jump on a call where everybody's just glowingly happy on a client call, you know? It's very rare. I'm sure you, our RStudio, you guys are used to that. I've been on the other side.
Sure, and I guess before, like, jumping off this topic of hiring, too, I know, Ian, we were chatting a little bit about, like, how a lot of job descriptions are written poorly in some cases, and I think I've also just, like, started to see, or what is interesting to me is I started to see different job postings that say, like, shiny in the title, or, like, tidy models engineered to get people interested, and was curious, Ian, if you could jump in and speak to that a little bit, too.
Sure, yeah, and that's something I encountered a lot. Recently being promoted into a leadership role for clinical data science, what we do in clinical data science is pretty specific, actually, and so going across different job ads, job boards, seeing different job postings, there's always some disconnect between the data science hiring manager and the recruiting partner, as well as how the job ad is written, and so, you know, one good example brought up earlier today and brought up in earlier sessions, like with Julia and then with John Thompson a couple weeks ago was, you know, and just mentioned earlier, the idea of Python and R together, interoperability, and that not being stressed enough in the job ad would get past the recruiter into the data science hiring manager, and, you know, they really just want Python, but they had R just as prominent in the job description with Python, but as a primary R programmer that I am, primarily what we use on the clinical side, that now, you know, we've wasted a recruiting interview and a hiring manager interview in something that could have been weeded out through the job description alone, and that's just my take on it, but that was a really reoccurring thing, and it doesn't have to always be Python versus R. It could have been, you know, they really want someone that is highly skilled in reinforcement learning only or primarily, whereas, you know, I stay into supervised and unsupervised learning mainly with the clinical stuff, so it was those kinds of things that I ran into.
Was it more so that the recruiter was then kind of weeding out for that, which wasn't in the job posting, or once you actually were talking to? Right. Well, that's hard to tell, and it's different for every job. Sometimes you don't get to the hiring manager. You're with a technical recruiter. You know, you obviously want to be honest and communicate what your skills are, and you say, you know, really, I'm a primary. I'm an R programmer, and I use it for, like Mitty said, I use it for stats. I'm not big on the data engineering, data architecture side. Being open and communicating about that with a recruiter is great, and you should do that, but that doesn't mean the recruiter necessarily knows how that does not translate to what the data science hiring manager is looking for.
How does that work in your team, Moody? Yeah, I was just thinking through it. It's not really very pretty either. Like, yeah, I mean, I don't have a solution, and we try to do the same. We have the same problem. Like, we end up on the recruiter has a sort of an HR type of call, but by the time the resume hits, it's basically almost a buzzword match of what you have, and then basically they offload it to a first line of defense, which is like some person in my team would basically have an interview, and then we ask the questions that you were asking in the beginning to try to fish out where the applicant stands, and then if they graduate from that, then they go into the sort of second round and third round.
Yeah, I don't like it's stuff, right? Like, especially in like my area, because like, well, I mean, I guess clinical studies is pretty technical, too. Like, you know, sometimes I don't know how to articulate it myself on a, like, so you're either, like, I try to, what I try to do is reach out to, like, the network, like, obviously folks in our RStudio, folks around that I know, and try to say, hey, because they're more than likely will have applicants that at least understand the domain, so it makes it a bit easier to kind of weed it out. But just generally posting, it's just basically, unfortunately, brute force. You just go through it, and I don't really blame the applicant or the recruiter. I don't expect the recruiter to know it as well as you, Ian, for example, right? And the applicant is also kind of in this state, it's a numbers game, right? Like, you know, so they will try to add more
things to the resume just to kind of get that first round. So, yeah, it's stuff. I think conferences, when they were more in person, events, those sort of things were a good way to recruit, right? So, yeah. I think we talk so much about some of the non-technical skills that we all look for, and then we talk about how people can't get past the first stage just because of a resume or recruiter, like, pushing down an applicant. But I'm just wondering, like, how, what's the best way to communicate those non-technical skills, then, to get to that next stage? And I think, Christian, that comment you made is really important here, too, where you're saying maybe you said you used to think anyone that was smart enough to learn the math side of machine learning, the business side would be easy for them, but that's not always the case. How do people show that as they're applying?
I want to jump back in, again, like, the point that John Thompson made a couple weeks ago about his team was the idea of, and these are kind of shortening his words, you know, he was giving this idea of we have great data scientists that do really great at tuning hyperparameters, but we have other data scientists that are really great at EDA and doing the program management side. And so that idea of taking the project from start to finish as a team model versus that idea of people jump in at different areas I think is also worth part of this discussion about recruiting, because am I recruiting someone to take a project from beginning to end, or am I recruiting someone who is really great at tuning hyperparameters and all of those other different disparate parts of data science, but is not so good at the program management side? Do you see the differentiation I'm making? So, you know, he was talking about that, and that really struck a chord with me because that was, I think, an essential part of how you would hire and something you would communicate as a data science manager to a recruiter. You know, I'm looking for this data scientist that can take it from end to end, or no, these are kind of these different areas that we would look for primarily, but if they have the rest of it, that's great.
I 100% agree. I tend to look at my hiring as how do I get people to complement one another, and that I feel can absorb that information from other individuals to try and create a little bit of a network in a way, because it's very, very hard when you're looking for someone that has like very specific skill set, let's say in healthcare and R and Python or whatever, but I can find like slices of that and then try to assemble those slices in a manner that's going to like punch above its weight class, so to say, rather than like have like, you know, try to find the person that fits. That's my opinion, but I totally agree with that.
Managing stakeholder expectations
How do you guys explain that to your stakeholders? You have five data scientists on your team. The stakeholders want 15 different things. You have five people, and they say you're only going to work on two things because you can do two things really well because you have that complement of people on your team, right? We can do two things in the next two months, and then we'll do another two things, but they're like, no, no, right now, you should be working on all of these, right? And I feel like that's such a hairy conversation every time I step into it. Any advice there from anybody?
Yeah, it's like for me, like for a while, they would describe my role as stuff and things. So, you know, no, I mean, like, yeah, so like what I do, at least I tie it to the business. So, we're like, hey, look, there's this, you know, I'm a bit of a luxury now because I have like basically three, four products that are making revenue. So, I just say, hey, we need to augment so-and-so to support at least one of those two products, one of those four products, and do EDA and take something end-to-end. I tend to get more end-to-end just because I'm on relatively, at least early on, I'm on much tighter timeframes. And then we have folks who are more senior that they can go into hyperparameter tuning, like check basically other person's work. But yeah, the stakeholders, I keep trying to tie it to a business need, like a revenue-generating need. I mean, also partially because my side of business is revenue-generating, right? I haven't had much success in like pure, like, you know, like saying I need somebody just to think through stuff. I mean, as important as that is, I have senior folks on the team who just do that, right? And they write, but the stakeholders generally don't want to hire, like they don't want to pay for someone just to sort of do research, right? So, that's kind of how I work around it.
I guess just to answer Frank's question, I think, I mean, in my particular situation, we do have a lot of things going on, but, you know, healthcare and our particular real-world data space, it's kind of difficult to find people that are like cancer experience and like our experience and epidemiology experience. So, it's like, these are like unicorns. If you find someone like that, someone probably also already has an offer out for them. So, we have to kind of augment. Otherwise, we're going to sit spinning our wheels for six months and you're never going to get anything done, right? So, I think in our case, it's a relatively easy sell to be like, look, we're assembling the team to be able to deliver on a variety of these projects. And generally, I think, in at least my industry, it's kind of understood that we can't have someone that does and do everything.
Yeah. A data analyst is not like every single data analyst isn't the same or every single data analyst isn't the same. Yeah. And that's the tough part. The stakeholders don't understand that necessarily. Like it's, they're usually on the business, like they're much more senior object, like peer management, like, you know, upper management. So, you got to tie, like, for me, I learned the hard way. You got to tie it to, you know, X percent is tied to revenue generating, Y percent is tied to sort of, you know, R&D and, you know, Z percent is tied to something else. Like, you kind of play a bit around that and then you get like, basically, someone says, okay, sure, that makes sense. Or, you know, but if you just say, hey, I just need someone to do research or, you know, or just to do some full data or things like that, you know, like it's not going to, at least on my side, doesn't work.
Selling the value of data science internally
Maybe shifting gears a little bit, I see there are a few questions from earlier that we hadn't covered yet. And Christian, yours was about kind of selling the value of data science internally. I was wondering if you want to kind of expand a bit on that question.
No, that was, it wasn't really a question for the group, right? That was just more of like, so, there's like the AWS, like, managed instances you can set up for RStudio. You can have it just like prepackaged to hit run. Those exist for RStudio Enterprise, but there's nothing for, like, the free version of RStudio. If I wanted to quickly spin something up and be like, hey, look at all these cool things, and then take that to my management team, I can't do that, because there's not that free version. I don't have three days to get an instance spun up myself when I could just do it at the click of a button. So, if there was a free version I could use to pitch to other people, that would be awesome.
I can actually answer that one. So, there actually is something that you could use, and I don't want to make this call to CLT at all, but there's a hosted evaluation instance that you could try out. And I know for some people that's been really useful for just, like, bringing the examples to, like, their meeting and saying, like, look, this is the shiny app I made. Here's the Jupyter Notebook I published. So, I can send that to you, Christian. And if anybody else ever needs that, you can let me know. I can always connect you with the right person, too.
But, Moody, did you have to do any of that internally, kind of, like, selling the value of data science, or was it already there and understood by the teams that you work with? Yeah, I mean, yeah, no, I had to do a lot of selling, like, both on the language and the value proposition. I mean, it was funny because they, like, yeah, data science as a term, and, like, we created teams after that, that also came from another acquisition we had. So, they kind of, there was a bigger top-down sort of initiative to scale up. Like I said, technically, I'm not data science and SMP. I'm product management, but I happen to have data scientists report to me, right? So, but, yeah, selling the value proposition there was a little tricky. And I think it's still tricky because some of it, some of the more, like, you end up basically trying multiple things and failing sometimes. And the business doesn't always like that, right? They want you to, they want to buy data, do something with it, and then make money off it. That's basically what we do, right? And basically, buying data and then finding out it doesn't work is usually, like, you must have done something wrong, right? So, I think that's part of the, I mean, I have less of a problem doing that. I mean, early on, what we've done, what I did was, you know, we actually presented to a pretty high up on the food chain. I still remember, like, our markdowns when they first kind of came out. And we were lucky that our CEO at the time was, he's still there, like SMP Global, Doug Peterson, basically. He basically was an analyst at Citi back in the day. And so, he looked at the sort of almost financial research report, and it was very interesting to him. And I was like, oh, wow, I can do this now? Like, you don't have to, you can actually pull the data, do the model, do it all in a markdown, and have the analysis all written in one simple thing. And I was like, I'm not saying that sold everything, but at least sold my team.
I'm not saying that sold everything, but at least sold my team.
So, I mean, it's a little harder now because, like, folks are, the threshold has gone up, I think, a lot more because I've seen, but, I mean, my suggestion to, like, selling to the business is always more like, oh, the value proposition of data science is, again, like, don't just throw, like, 10 different R&D efforts out there. Pick the one or two that actually succeed. And then for those ones, sort of do, like, what I do with the incubator, like, build a Shiny app and do it the way you think a client would do it, and then show that. Because that sells it more than just saying, I can do, you know, you know, a soup of basically technical jargon, right? That doesn't help anybody.
That's a great point. Ian, I see you unmuted, so I take that. Yeah, well, you know, and Moody was speaking to revenue generation, and part of what my team does in my experience is actually cost avoidance as well as clinical outcomes. And so the way I try to do the value proposition to leadership and the way when I've been under higher data science leadership in previous positions is a simple calculation of what's called blue dollars in cost avoidance, where we really just try to keep a high level of, we're going to go in, we're going to do an internal consult for a team. If we can do this consult and save FTE hours, you know, we're going to collect those blue dollars and present them quarterly saying, you know, we saved this many FTE hours by overhauling a process, avoiding costs, getting better clinical outcomes. So that's kind of a summary on how I do it on my end. It's a little bit different of an approach. Yeah, we have that in the content team, and the other data, that's exactly what we do. Efficiencies and, yeah, I mean, I don't do it, but like another, my colleague, he runs, exactly, that makes sense. Yeah. Cost savings.
Most unique problem solved with data science
Moody, what has been one of your most unique problems across most industries that you've solved with data science? It was an anonymous question that came in there. Most unique problem? Well, I would say the China sentiment. I really knew nothing about basically China, like I said, you know, beyond the news and stuff, right? And that was like, took about three, four years to get to the point we're at now. And I like, and the funny thing is none of my team members spoke Chinese. So, and we had to basically build something using the 20,000 or so companies. And the amazing thing is there is a lot of data that they collect. And because we're SMP, we can access that data. And in some cases, a lot of it is public domain, but actually trying to sort of understand that business and that context and make it easy for the rest of the world to sort of have some sort of window into that area, especially when I really didn't know as much or anything really. That was quite, it's very idiosyncratic, right? And then we hired, obviously, labelers. We had some folks who are in China who specialize in that market, help us. So that was very, very unique than what I'm kind of used to doing, right?
Kevin, I'm seeing now that that was your question, but it was coming through as anonymous. So feel free to jump in if you wanted to add anything there as well. Yeah, I'm glad I got the chance to hear the answer to that question. Yeah, I always hear
very industry-specific results on that question on which is the most unique. And it's a nice perspective that you just gave. Yeah, basically, if you generalize it, it's just that situations that are very idiosyncratic, that happens to be the most idiosyncratic one I think I've done in about the last 10 years or so, right? You actually have no idea. So there's other things that are more like just problems that are more like big data problems. Like I'm working on something on transactional data with credit card payments, just like terabytes of information that you've got to mine. And then that also presents its difficulties, right?
But yeah, like something complex. Again, a lot of it was spent understanding what we were trying to do and for whom. And then from that point on, it became kind of almost a linear set of instructions to basically execute, at least like almost a decision tree so you can exclude certain options. And I learned the hard way that you see a lot of like those prepackaged NLP tools out there, they are not really, they're mostly on the sort of the English, Latin sort of side of it. Like that area, there isn't really anything that, we use Fastex, for example, and that was really bad actually for finance and simple white Chinese. Because they're trained on a lot of Wikipedia data. And when you start putting in the language plus the context, they kind of fall apart. So if you're just doing something that's sort of generic, hello, you know, kind of NLP type of problem, it's fine, right? But once you start actually trying to use it to do something more specific, it falls apart. And then we had to do our own basically vectorization for it.
Project ingestion and data readiness
I see another question from earlier around this project ingestion process you have, and it's quite a longer question, Rahul. So I might have you explain it, if that's okay.
Yeah, sure. The question is really on how you deal with project ingestion, right? As data scientists, we get multiple requests for projects. And then all of them are urgent for the business. And when we get into it, that's when we find out that the data needed to make the decisions is either completely missing or partially missing, or not enough to make the decisions. But that's only several, sometimes several months in, right, where our data scientists are spinning their wheels, trying to like, you know, put things together, this thing in a Postgres database here, there's an API there, there's like PDFs, you've got PDFs sometimes that we have to scrape. And it gets frustrating because a lot of data scientists really want to focus on the modeling aspect. And we all recognize that 80-20, you spend 80% of the time on the data. However, there has to be the balance where if that's all we're doing, and all the time we're doing is that's all we're doing. And, you know, all the data scientists are very capable, right? So just because you can do the data cleanup, and you know, data munging, that's not your primary role, right? So then, you know, folks do get like burnout. And so we are trying to develop a better ingestion process for how do you accept projects and how do you evaluate data readiness before we engage the data scientists and launch it formally as a data science project. And I was wondering, do you guys have any thoughts around that or any existing processes you follow?
I'd be interested to hear. Yeah, so like, yeah, like, like, again, because I'm running a small team, my opportunity cost of someone failing, not because of them, but the data is actually high. So I don't have a like a bulletproof way to sort of prioritize projects. I mean, generally speaking, we try to look at revenue potential. Is the data completely new? Or is it something that was new? And is it new and can be linked to something we already have? Generally, if you get something that's sort of independent from the rest of the database, like your value proposition is going to be much harder. But if it's something that is related to some of the companies we cover today, and they have other measures, then there's a higher success chances. But yeah, it's happened to me where you basically work on something for like, six months plus, and then you end up with like, well, I guess it didn't work.
It's unfortunate, but I mean, it kind of comes back to the business side, because you kind of try to whoever's adding this project, you try to do your best to have them explain in very simple terms, what they want out of it. Right. And that's where I kind of use that sometimes when things kind of fall apart, they're not very well defined. And it's like, well, we really didn't know. So we tried and it took a while because of these complexities. And at least we found out that why we failed. So that's not really an opportunity anymore. So that way, you kind of dance a bit around the issue, right? But yeah, it happens, happens a lot. But yeah, I also thread the work through other teams. So depending on like, so sort of lower value things that I kind of give to other folks, see kind of monitor it. So you try to kind of, again, at the end of the day, you want to really support the business line, and then they're bringing it to you, not because it's just kind of nice to have some reason behind it. So it's good to sort of try to support them as much as you can. And burnout, I just put a lot of folks on like, at any point in time, any members working on at least a project and a half, or maybe two. So like, you know, they're taking one end to end and then the other whatever residual is worth supporting someone else on my team so that they don't get burnt out too much. Right? Yeah.
Dashboard maintenance and handoffs
The other question I have is, is we sometimes are falling victims to our own like success and capability, because the initial phases of a project are often very descriptive in nature, right? We have, like, we deal with a lot of engineering data that engineers are not looking at. Right. So to build, you know, we focus on forecasting models and getting those insights. But the very first part is just all the descriptive analytics that has to happen. And then, you know, we then end up creating dashboards and show them that, hey, this is your data, these are, you know, like the visual insights before we even touch the modeling, which they love. But then we also end up, like having to maintain those long term and be, you know, end up becoming the dashboard maintainers, right? Which I see the value in it. But it, you know, making a one time dashboard to show insights and then focus on the modeling is very different from now maintaining a production grade dashboard, right? That if anything goes down or the upstream data changes slightly, suddenly like this, you know, you know, drop everything and fix it, because there's so many people looking at it. How do you face similar issues? And do you have or do you have teams that like, take over dashboards? And now they're, you know, that's their job is to just maintain these production widgets?
Yeah, no, yeah, that's funny. It's like, yeah, no, yeah. So, so, like, so, like, I tried to, and then this has happened over time. So it wasn't like, I didn't really think about it, and then structured it, just fell into problems and then tried to address it. Like the client facing part, the incubator I own, so that I'm kind of tied to. So, you know, so I don't like, we try to write it, we have some IT folks sort of help, but we kind of, now the internal stuff, generally speaking, once there's some success, and there's no way to, like, there has to be a way to production, I turn it over to the, like our tech team, obviously, with the product, the segments, product manager, and then they work together to get it done. And then I shut down basically, the old app has a lifetime. So I don't, I'm exactly not on the hook, basically keep maintaining it. Yeah, and sometimes like, what's more like, so in my world, it's like, I'm kind of a bit faster than the general data science team, and the data science team is a little bit faster than the tech team, and the tech team is more like business as usual features. So where I've basically been kind of struggling is more transitioning to the technology folks on things. So, you know, you do two, three releases of something in production, because we wrote some service endpoint that's basically supporting it. And it just happens that the software engineers don't necessarily have the same background. And I'm talking programming background, not math. Right. And that's kind of where I'm kind of focused on now is getting that team up to speed so that they can actually not involve my team and feature enhancements.
But yeah, I mean, you know, like, what I end up doing is kind of just allocating folks directly to them and saying, you're owning it now after this release. So you have to sort of, we'll hear to, you know, you have to shadow, get your hands dirty, and then basically take it over eventually. And anything that has a path to production that's already defined, I've gotten the feedback, I shut down, basically. Yeah. How do you handle that, that handoff then, and getting all the teams on the same page? Yeah, so I mean, it's obviously, I have a counterpart in the software engineering team, right? And there, we agree that that's the right thing to do. And then it's more like, how many releases does it take on if we need to resource the software engineering team on our end? So we do like the first, like, obviously, like, it's almost like, what, like, the first time it's like, you know, 75, 25, 75, my team, 25, software engineering, eventually it's 50, 50, and then 25, 75, basically, so two to three releases.
Even that's a little slow for my taste, but like, you know, but I mean, it's just the, like, like a lot of the more, like, programming language .net, JS, JavaScript folks, right? Java, they're, they're just not aware of, like, R and Python. So it's more of that side of it. Then it is like, again, it's not like, the models we end up owning anyway, there's no way around that, right? But like, but I think it's more like a skill set mismatch, right? But, yeah.
An impressive product: the financial extraction tool
Thinking of those, those initial deployments to where you're maybe presenting something to the executive team, what's something that your team has worked on that really impressed the executive team and delivered a ton of value? Oh, the, the, we have that, the financial extraction tool I was talking about in the beginning, where basically you take an unstructured PDF of a small business or a private business, and you upload it into our system. It's like, literally sitting, like, you just got it from an email or someone else. And then in a few minutes, you click through a few points in the desktop. And then all of a sudden, that data that you care about, those financial line items across the balance sheet income cash flow are already in our platform with assignment, with the mnemonics, like with, with the metrics that we create, our company hierarchy, our taxonomy, in like a few minutes, that's there, right? That really resonates a lot with basically the, the business. And, and that's like, kind of an ongoing sort of product that we're, we're, we're expanding further and further into.
It's a very simple message, right? Like, you've got something that came from someone else. And then all of a sudden, it looks like it's in your desktop. We have, we do this in a systematic fashion for our public company data, fundamentals data, right? That's like, lots of teams, validating, checking, NLP, sort of extracting and all that, right? But it's not like, in a way, it's organized, there's a much more rigid pipeline here, you're talking about something, you know, you just emailed me, and I can just upload it as long as it's a financial statement. So that, that message is simple to understand and resonates.
It's a very simple message, right? Like, you've got something that came from someone else. And then all of a sudden, it looks like it's in your desktop.
And, and I'm, I'm sure other people might understand from this, but what, what exactly is that deliverable? Is it like a Shiny application or somebody? Well, the prototype was in Shiny, like, you connect back, but now it's in our desktop life. So yeah, you're getting basically like a financial statement of a small business. And then you're taking out like total revenue, net income, EBITDA, like all the measures that you care about, but you don't actually have to highlight or do anything. You just upload the document and it'll extract out the line items and then takes those line items and assigns them to our, like, taxonomy, which is an SMP, you know, over decades has been built in this taxonomy. And so in a few clicks, you're basically take something that never resided in our platform, and all of a sudden it looks like it's part of our platform.
And then you can basically, then you can do credit scoring. So after that, you have those numbers that you can save and archive that information. You can do credit scoring around it. You can do, you know, a lot of other, build a lot more ratios around that. So it becomes as though it's a company that we've ingested normally. But the user is completely user-driven. Like we don't touch it. We don't even know what the document is or who the company is. It's proprietary.
Oh, wow. And I know an hour goes by really quickly and we're already at the top of the hour. So I know some people have to jump off and Moody, if you have to jump off, let me know. But I'd love to stay on and ask a few remaining questions if that's okay. But for anyone that has to leave,
Moody, what's the best way to get in contact with you if people have follow-up questions? Is it LinkedIn? Yeah, LinkedIn or probably LinkedIn is the easiest. I can track it better than my day-to-day email. Yeah, LinkedIn is good. Perfect. You know, yeah. Rachel, maybe I can send you some of the folks we're hiring and then you can share it with the folks, like the everybody on the data science hangout. Definitely. Some URLs. Yeah. Cool. Yeah. Send them to me and I'll put them up on LinkedIn as well.
I do see there's another question that Frank had around a podcast. You said it comes from a podcast that you've been watching, Frank, or listening to. Frank, are you still on? Yeah, I didn't want to take credit. I think it's a great question. Moody, it sounds like a lot of the work that you and your team does comes from stakeholders. So I'm curious on the flip side of that, if you had unlimited resources, what is the problem that you would try to tackle? Unlimited in- Nobody's asked me that. It's so freeing.
That's true. Yeah, that's a good question, actually. Yeah. I think there's a, like S&P, like we, I'd love to basically dig into our full data repository and identify what can be linked efficiently to kind of target, like we have so much data that we don't mind. Like the China sentiment data, for example, that was, somebody else bought that a long time ago as part of something. And it was just sitting there on a folder. So I would love to sort of do more projects like that, where I would look through our full repository and see what we actually house today without needing to acquire additional data sets. And then seeing if we can kind of combine what we have today to something more useful.
Do you think that that sounds like a S&P global data knowledge database or knowledge graph? Yeah. So we have data lakes, like we have a lot of data lakes, a lot of data warehouses. We have a basically structural way of like keeping track of it. The problem is it loses that business context, right? So it becomes kind of a, one segment buys the data, it gets linked, and then that segment uses it, and then we forget about it. And it just sits there, and there are obviously technology people and data people supporting it. So I would just kind of go on a fishing expedition, basically, go through it and just see what can come out of it. Because I would hazard a guess, we're sitting on a lot of things that probably cost us money to maintain, and we just probably use a fraction of its value.
things that probably cost us money to maintain, and we just probably use a fraction of its value.
But I mean, that's the nature of the business, right? Being kind of like an analytics, you're always interested in the new data you just bought and how it links to it. And then eventually, the new data becomes old data, and so you forget about it, then you move on. So I think that would be it, yeah.
Resources and recommendations
Frank's question made me think to ask this as well. Moody, are there other podcasts or resources that you listen to or read that you'd recommend to others? Not podcasts, but like, yeah, I mean, I read a lot of our RStudio posts, like Jonathan Regenstein's posts on LinkedIn, right? Like Matt Bancho's, a lot of that work, like beyond the usual, like Wall Street Journal kind of stuff. Podcasts, I haven't really been kind of following up too much, to be honest.
Trevor's question on the development process
Trevor, I see your question from earlier, as well, and I see that you're still on. Would you want to ask that question live and introduce yourself? Yeah. Hi, everyone. Hi again. So I'm Trevor Thompson. Always, always try my best to make the meeting. So I'm actually a research manager in Jamaica, IGT. It's a global corporation. We have a big operation in the US, also in Italy. We're in over 100 countries, right? So I do research. It's a gaming company. We provide support for lots of companies around the world, both technology and also marketing, right? So we do a lot of research.
What I wanted to know, Muli, because you're kind of unique, where your product seems to be a software. You know, I was just trying to see how the approach that you take to developing applications, because as data people, be it analysts or scientists, I'm always curious about the process that we use to come up with applications. It could be a shiny application, but what process do we think about when we're going through? Do we have a systematic way of developing what we do, or do we do it in an unstructured way? That's the question.
Yeah. So typically, we'll have like some idea that there was a customer problem that's being faced. And then it's like, okay, so we take that idea, refine it into some form of at least an easy to understand, simple language business problem. What are we targeting to solve? Then we go back and hunt to see if we have data that can solve that problem. And if we do, we build like a mini prototype around it, right? And that involves basically that includes the model. And then we push out something kind of rudimentary to begin with, at least for internal folks to understand what the prototype looks like, refine that over some time, and then push, like graduate that prototype into something that external folks can use, get their feedback. And that can be done in Shiny, right? get their feedback. And then eventually you've reached some happy medium.
And that's what you end up basically building as requirements for the technology folks to build in our platform. So that's generally the process flow like end to end.
So yeah, you'd basically try something first, and then you kind of show it to whoever asked you. Okay. Yeah. Okay. So you do a prototype. Okay. Yeah. Shiny is really good for that. Like if you can, so, so long as you can, you have an idea of what the requirement is, typically you'll, you'll build a model and then build it in Shiny and then show them something quick. Even on my desktop, I don't need to use like the biggest thing. It's something simple. And then at least that's sort of your first litmus test.
Yeah. I would say it sounds as if you are using something like loosely speaking, agile technique, because you, I guess you have different iterations. Do you say you develop a prototype, you get feedback, then you make adjustments and the cycle continues? Yeah, it's not a waterfall approach, right? There is like a more agile, safe approach, right? And then, but the technology team are much more on that safe, safe, agile sort of approach than I am, but we try to follow sort of the same methodology. A lot of, a lot of small iterations, right?
ROI of open source languages
Ben, I see that you're unmuted. Do you have anything to add there too? Or feel free to put it in the chat too. Moody, there's just a few more remaining questions that I see coming in on Slido. And one is, are you able to estimate the ROI from using open source languages? If yes, do you have a clear business example of use cases where you saved money by using them? On using open source languages? Right. Yes. So like R, Python, that level? Or like libraries? I know we couldn't assume, but I'm going to assume that yes, R, Python. No, it's funny. No, I got it.
like languages you, yeah. So I had, like, nobody asked me that. So like, but a few years back, I had a question kind of similar to that for internal purposes. And what, it wasn't even my idea, actually, someone else, like what we did to measure, what he did to measure it was to use lines of code. So if you had written something in C++ versus, say, R, how many lines of code, less lines of code you did in R versus, say, C, right? And that kind of gives you an idea about your efficiency as a programmer, right? And that kind of takes over also the library part, because in R, you're still probably writing using some, you know, some other libraries, Quantlib or something like that, right? So that usually, that measure kind of shows the value proposition. Now, how you tie it to revenue, that's a different story. It was more time-saving than it is revenue.
But yeah, no, not like, I mean, you have that MATLAB versus like R, Python kind of question around, I suppose that's, but that's, I don't know. We have folks who use MATLAB and I use MATLAB back in the day, right? Like, there's nothing wrong with it. I just, you know, there's room for, I think, a lot of different languages depending on the application. So, you know. It sounds like a lot of the value that you've shown has come from the data products that you've built with languages. Yeah, just generally, like, I mean, don't go crazy, right? Like, you know, Julia, you've got like a lot of other stuff, right, out there. I mean, I'm sure they will have some applications down the line, but yeah, like, at least like R, Python has kind of become much more business as usual. I mean, a few years back, R wasn't, I mean, not a few years, like, you know, more than five years ago, right? Like, but I think it's much more accepting now, right?
So, but yeah, because you're, you know, again, you're, it's your, like, your maintenance overhead can go up. So, you might, you know, you have to, there's some, like, intuition you should have, right? Because if someone, like, I hire somebody who knows Julia really well and then builds some amazing application and quits, and then I've got nobody left, right? So, you have to have that sort of, there are checks and balances around what language you use. You shouldn't just let everybody do it, but yeah.
Tech stack
And taking that a little bit further, I see there's a question around the tech stack that your team uses to do this work. And maybe I'm combining two questions here, but also thinking about that tech stack and working with certain clients, do you build back to, like, certain groups within the company? Yeah, yeah, we could work closely with the data science team who are under technology. So, that's, like, that's very helpful because we, again, kind of have that checks and balances around. Like, tech stack, I mean, we use R and Python quite a bit, right? I think more like Redis, MongoDB, depending on the situation, like, obviously, obviously, a lot of our internal, we have a VPC in AWS, so a lot of sort of cloud DevOps that knowledge there, Spark for distributed programming, trying to think what else, like a SQL, like a lot of SQL, depending on Snowflake, we've been using Snowflake quite a bit, like, we've been, but we still do a lot of relational stuff, like we have Oracle and SQL Server, so a lot of, like, knowledge around pulling the data and how to sort of do it. Databricks, we've been a lot of success in using Databricks as a platform for production pipelines. I think that's pretty much it, I would say.
Advice for aspiring data science leaders
Thank you, Moody. I'm checking to see if there are any questions that I missed, and I am not seeing any, but I want to do a quick check to Rob and Rob, who've been helping monitor as well. One final question that I always like to ask, and I think that a lot of people on the call may also be in this position, but as aspiring data science leaders, what advice would you give to someone for maybe getting to that leadership role?
Yeah, I mean, we kind of hit on it earlier, but like, yeah, I mean, really understand the context that you're trying to solve for, right? Like, I mean, obviously there's some level of technical capabilities you need to have, right, both in math, statistics, and programming, but after that, to move up, you really need to understand what the business use case that you're trying to solve for is, and in very simple terms, right? And that's, I think, what makes it sort of a quant versus sort of a leader, right, is understanding that context, and, you know, and you should question your own numbers a lot, right? Like, there were some, a lot of times, like, it's like, oh, look, the correlations are like this. It's great, but if you see something that doesn't look like the norm, question why, and then dig into it, and you should be ready to explain why it's not like the norm. I care more about those outliers than otherwise. So, like, it really is kind of questioning your numbers, and then understand the context that you're applying it to.
Not podcasts, but like, yeah, I mean, I read a lot of our RStudio posts, like Jonathan Regenstein's posts on LinkedIn, right? Like Matt Bancho's, a lot of that work, like beyond the usual, like Wall Street Journal kind of stuff.
Awesome. Well, thank you so much, Moody, for joining us, and thank you for kind of running over with us past the hour. I really appreciate you sharing all your insights with everyone and just being so open. So, thank you to everyone for joining as well, and hopefully we'll see you back next week. So, same time, same Zoom room. Thank you all. Have a great rest of the day.