Resources
Dec 27, 2023 |
59 min
|
451 views

Matthew McDonald @ KBRA | Data Science Hangout

We were recently joined by Matt McDonald, Senior Managing Director at Kroll Bond Rating Agency (KBRA) to chat about “punching above your weight and running a small, effective data science team.

Speaker Bio:
Matthew McDonald is a Senior Managing Director at Kroll Bond Rating Agency, responsible for managing the Quantitative Modeling team. Matt joined KBRA in 2015 to Build out KBRA’s Model Risk Management framework. Before joining KBRA, Matt held various modeling roles at GE Capital, IBM Global Financing, priceline.com and PriceWaterhouseCoopers. Matt holds Masters degrees from Columbia University and the University of Connecticut, and a BA in Mathematics from Colgate University.

___________________

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc

To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)

Thanks for hanging out with us! 💛

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Welcome to the Data Science Hangout. Happy December, everybody. If this is your first time joining us today, so nice to meet you. I'm Rachel. I lead Customer Marketing at Posit. If it is your first Data Science Hangout today, I encourage you to say hi in the chat so we can all welcome you in here to anybody joining for the first time too.

The Hangout is our open space to chat about data science leadership, questions you're facing, and getting to hear about what's going on in the world of data across different industries. So we're here every Thursday at the same time, same place, unless it's a holiday. But if you're watching this recording on YouTube later, the link to add it to your calendar will be in the details below as well. We're all dedicated to making this a welcoming environment for everyone. So we'd love to hear from everybody, no matter your years of experience, titles, industry, or languages that you work in. It's totally okay to just listen in here if you want, or maybe you're out on a walk or on a lunch break or whatever time of day it is for you.

But there's always three ways you can jump in and ask questions or provide your own perspective too. So you can raise your hand on Zoom and I'll keep my eye out here. You can put questions in the Zoom chat. And if it's something you want me to read out loud instead, just put a little star next to it or asterisk so that I know. And then lastly, we do have a Slido link where you can ask questions anonymously too. And our co-hosts here in the background will share that. Just another quick note. I forget to say this sometimes, but I love to see the conversation that happens in the chat during the Hangout. It's so cool to see all the resources that are shared there. So I encourage you to use the chat during the Hangout as well and share all that you want there. It's fun to read through it later.

But I'm so excited to be joined by my co-host today, Matt McDonald, Senior Managing Director at Kroll Bond Rating Agency. And Matt, I'd love to have you kick it off with introducing yourself and maybe sharing a little bit about your role in something you like to do outside of work.

Oh, okay. Yeah. So I need to, for business purposes, make the following statement. So the views and opinions expressed during this presentation do not necessarily reflect the views or positions of KBRA and its affiliates. So that's what's going on here. All right. So thanks so much, Rachel, and thanks to everybody for joining today. And thanks to Posit for hosting the Data Science Hangout. So my name is Matt McDonald and I live in Stamford, Connecticut with my wife and daughters. And I tell people I'm a statistician when they ask me what I do. And I've been involved in data science for quite some time, coming up on 20 years, I would say.

I work for Kroll Bond Rating Agency. So I think we rebranded recently. So we are now known as just KBRA, I would say. And so what does KBRA do? So we provide credit opinions. So we're a financial services company providing credit opinions in the form of credit ratings. We've been around since 2010. So we are a post-financial crisis credit rating agency. And our competitors are companies like Moody's, S&P, and Fitch. So these are companies you may have heard of. I've been at KBRA since 2015. So I just surpassed my eight-year anniversary. So I've been here for quite some time.

And when I started here, I was in charge of model risk management, which is something that if you're in the financial services or if you work for a bank, it might be something that you're familiar with. I don't know how widely known it is outside of financial services, but there's a need because a lot of these companies that I've worked for in the past and that I work for now are regulated. There's a need for controls on the model. So we have people who are dedicated not to building the models, but evaluating the models. So I started my career here at KBRA in the model risk management space. I sort of created that framework and rolled it out in the company. And now I'm in charge of the quantitative modeling group. So now I'm sort of on the other side of the fence. And I'm in charge of building the model.

So I run a small team of data scientists, econometricians, statisticians, financial modelers. And our job is to provide models to credit analysts. So these are models that these people – no, these people – that are used in the credit rating process. And you can kind of like group them into a small sort of – you can categorize them. So a lot of them are like statistical models. You can think about classification models, logistic regression type approach. And other ones are like stress testing models. So lots of times we'll be looking at scenarios that sort of could happen but are bad and plausible. And we want to get a sense of what would happen to some financial portfolio or something along those lines. So we're looking to see when things get bad, how bad will things get, and how does that translate into our analysis of the credit.

Sometimes we get engaged to sort of help people sort of develop these scenarios, like what would an adverse scenario for interest rates look like. And we get involved in that. And then there's sort of cats and dogs type stuff around data analysis. So we'll be drawn in and engaged to help people analyze sometimes large data sets, sometimes not large data sets. I'd say that we're not really in the huge data space. We're kind of dealing with data sets that fit into CSV files that we can look at. So that's kind of what we're doing. Those are the types of models we're doing.

Team building philosophy

I realized that as I was describing this talk, I think I might have said it's about team building. And I'm not really going to talk too much about team building. I mean, you might see that I'm reading some notes here. So I'm trying to have some frame structure to what I'm doing here. But, you know, I mean, as far as team building goes, you know, I always follow the sort of guidance of like hire people who are good and are talented and that you like to spend time with. And so far, I've been quite lucky. So I have that largely where I am right now. So that's good.

What about something you like to do outside of work, too?

So I play guitar and I actually haven't picked it up a whole lot recently. But once in a while, I'll do like an open mic type setting type of a thing. So that's fun.

I said there's lots of musicians in here.

Oh, nice. We should start like a jam session someday. Posit data science hangout jam session. Yeah. Don't rely on me for singing, though. That's not I do it only out of necessity.

So someone asked, what kind of music do you play?

Oh, so I mean, I'm kind of more of a classic guy, but it's, you know, I don't really have the luxury of playing the music I like to play. It's more of I'm limited to what I can play. So, you know, three chords are pretty good singing talent. Never stop Bob Dylan. Nice.

Okay. One other fun question before we they're all fun questions, but a funny question. Last week, there were a bunch of people in the chat commenting on their favorite or worst office jargon, like boil the ocean or. Oh, yeah. Do you have a favorite or pet peeve one?

No, I mean, it's sort of it's like a bit ephemeral, like I'll hear it and I'll just sort of shake my head and say, like, oh, then I let it go. That's one of the things it's survival skill one on one. Let things go. You can't carry them around. I will say that sometimes you'll hear someone use a noun as a verb. So like when there's a perfectly good verb out there to be used. So someone will say, like, we need to solution this problem. And that kind of rubs me the wrong way a lot. There's a word called solve that would work perfectly fine right there.

An example modeling project

So funny. Well, I just was getting people going with that in the chat there. Matt, for some of us that are not in the financial space or maybe familiar with credit rating agencies, would you be able to just kind of walk us through an example project or something that your team might be building a model for? You don't have to get too into this. Yeah. Yeah. I mean, I can't I saw a comment in here about a particular rating that was sort of issued by KBRA. I can't really get into that. I was not I was not directly involved with that. Like one of my models may have been sort of employed for that. But that's one of the things that we don't really we're sort of a step away from from like that that that ground level.

So we'll build a model that's available for people to use. And part of our challenge is to ensure that people understand its limitations and understand when it and when and when it should not be used this time, when it should and should not be used. I mean, I we put a lot of thought into how we get things out into the world. And we're starting to sort of gel around around like a playbook that we employ.

So, you know, I'll give like just a theoretical example of like a model we may build. Let's say we wanted to to build a model that would give a probability of default, a classification model. So, you know, that would be like, OK, we're going to go collect data on some sort of on corporates or maybe banks or something like that. And we'll think about what types of things might be driving default. And then we'll sort of perform a statistical analysis and we'll build the model. And maybe that would be like, you know, the simplest, simplest thing, a logistic regression model. So if you tell me companies some things on their balance sheet, something about their cash flow and their and that type of stuff, then then maybe we could build a model that would help you figure out if that that that entity will default.

I mean, that's really half the battle. And that that's a huge battle, because, like I said, we're regulated and we have internal controls who are evaluating our work. So we need to really make sure that, like, all of our work is repeatable and transparent and inspectable and we can explain it and it's well documented. And then, you know, we have the expectation that there's evidence based. So we perform back testing and we present that.

But, you know, the the next problem and this is when I say playbook I'm talking about is like, how do we sort of make once that model is been built, how do we allow that model to be used by people, by real people who are doing a job? And and and, you know, we are, I mean, shameless plug for Posit. So we use the Posit Teams platform. Right. So that allows us to do our work in a common environment. And we, you know, we have controls in place that help us make things repeatable. We're using Git. We're using Renv. If we're building in R, then, you know, we're making sure that our environments are sort of repeat or we can build them up. And then, you know, we are.

I mean, we're not 100 percent there, but we've been experimenting with things like Vetiver and Plumber for deploying models via API that are then usable by people.

Bridging teams and breaking down barriers

You know, one of the things that we we sort of have been really working on and really focusing on is like establishing sort of ourselves. I mean, the conundrum that we have is that we're a small team. Right. And we kind of sit between. A technology organization, we're building software and and a in credit analysts who are doing like the work of credit analysis. Right. So like the tool gets between those two folks and between us are like kind of radically different. Like I would say the analysts are largely using Microsoft Excel for a lot of their their work. The technology folks are using Python mostly for like in a software development context. And, you know, I'm an advocate of using the right the right tool for the job. So we're using Python and R in more of a scripting context to sort of do our analysis and build our model. So sort of being able to talk to all of those different parties has been something that we've been really working on. And we've had a lot of success with that, with the tools that we've chosen in in sort of making breaking down some of the barriers that we face.

Can you expand upon that a bit, maybe for some of us who are having who are kind of in the middle of that right now? How do you break down some of those barriers between teams? Yeah, I mean, it's like it's a balancing act, I would say, between. You know, reaching out to people and operating on their turf,

you know, talking their language, but also working very hard to sort of retain some sort of line line of ownership of things. So, like, you know, I would say, like, when we do deploy an API, we've sort of had some success and done some experimentation with, you know, providing people with some code, PBA code in their Excel spreadsheet to call our API so that they don't need to learn, really learn a new skill to use our work. And like going the other way, you know, some of our projects, some of our models are not really that that sort of like regression based model where all I have is a set of data and I'm going to create a prediction. Some of them are really more software type models. They're more complicated. These are like simulation models, Monte Carlo simulations where we're going to that's a that's a pretty standard in credit risk modeling, like a pretty standard approach. And so, like, you need to build software to do that. And when we, you know, I want to I don't want to build own and maintain software. That's sort of a job unto itself. So we do engage with technology. And that's like when we go there, we've sort of decided that, like, they really don't speak R somewhat. So we will have to do more Python stuff and switch our mindset of how we're using Python from that idea of like performing some analysis, writing a script that does a thing into writing like Python code that is that is integrated in like a software development type of a project. So like, you know, I.

I'd say that we've been had a lot of success with, like, learning the tools that are employed by technology and trying to figure out how we apply those to our work. So, like, everybody on the team has been learning Git. We really are getting very good at Git and we find that Git is a super useful tool, not just for, like I said earlier, we need all of our work to be repeatable because we could be sort of engaged by a whole bunch of people who are asking us to explain what we've done. So Git is super helpful for that. And all of our code is like transparent, inspectable. People can see all the changes that have happened over time.

But also we're using some of the project management tools that they have. So things like JIRA and Wikibase documentation things are we've had a lot of success with. Thank you. I see, Julia, you had a follow up question here on the repeatable work. Do you want to jump in? Sure. Thanks. This is really interesting.

So you were just touching on how you're working to be like replicable and transparent. Does that extend also to your models? Do your models need to be explainable or is it OK to use black box models as long as the process to derive them has been transparent in your industry?

Yeah, no, I think that there's no like regulatory need or sort of like nothing dictating what kind of like algorithms we can use to build our models. But I would say we definitely sort of stay on the side of simpler, you know, maybe maybe slightly less performative models that are explainable because it's more of like not only do I have to explain it to the credit analysts, but like they they probably need to explain to like, you know, a lot of other people how they arrived at their decision. So if we have just an absolute black box, that doesn't really work for us. Although we have like used some of those approaches to help us like benchmark our models. So, you know, when you, you know, you have kind of like a matrix of like model performance and and interpretability, I would say, would be kind of a graph I've seen in the past. And we're kind of like on the interpretable end of the spectrum. But, you know, lots of times we'll ask the question, like, what what are we leaving on the table as far as like model performance when we make that decision? So we'll like employ like a more black box type approach to say, like, look, if we didn't care, we could explain this to anybody ever. What what could we achieve? And we kind of use that as a in our decision making process.

we definitely sort of stay on the side of simpler, you know, maybe maybe slightly less performative models that are explainable because it's more of like not only do I have to explain it to the credit analysts, but like they they probably need to explain to like, you know, a lot of other people how they arrived at their decision.

Thank you, Andy. I see you had a question a bit earlier. Do you want to jump in here? Sure. Thanks. Hi, I'm Andy Nunez Brewster. And my question was about the looking ahead. If you're using you did answer the question, actually using some simulation model versus just data driven exploration. And I was curious if you're doing that in R or kind of what's the stack on that side of the house? Thanks. Yeah. Yeah, I would say like the we I mean, we use both Python and R for our sort of more backward looking statistical data driven models. And then, you know, those models that are more like simulation based or Monte Carlo type things, just simply because we are. You know, we're eager to sort of tap into some of the the good stuff that our technology team has been building. So we are we're primarily if we're going to be writing software aside from like internal R packages that are we're that are really like my team is the audience for those where we're going to build a package that's going to be help us do something. You know, we'll use R for that. But like if we're going to be building software that's going to be deployed. Across the company, then we'll be looking to Python mostly.

You know, some of the other some, you know, like that's, you know, we've been experimenting with like the very simple and straightforward, you know, I think I've heard the term microservice, but I probably am misusing it like the idea of like, I have some data and I want to get a prediction from that data. Like we've been working with, like I said, Plummer and and Vetiver are those types of things where we can deploy that and those get deployed to our our Posit Connect server. You know, the software stuff happens mostly as sort of a collaborative effort between us and our technology sort of folks in Python. But another thing that we've had a lot of success with is when we're doing like more of an analysis or helping somebody sort of maybe we've actually written a few research pieces, mostly internal research pieces in we've been using Quarto quite a bit. So Quarto is a very, very useful tool to create documents that are sort of very closely interwoven with the underlying data.

And that's been a super useful tool. And that kind of gets us out of the business of emailing spreadsheets and Word documents and that type of stuff. Like I much, much, much prefer the pattern of publishing a document. We publish it to our Connect server and sharing a link with somebody. And then if I need to update it, I can update it there. And there's even some really nice features that we've we've seen about like you can have the same code, the same Quarto document generating the web page that someone's looking at and downloading it as a PDF. And the PDFs look really nice. So we've had some really good success there.

Learning Git as a team

Awesome. I see there was a question about Git a bit earlier here. Let me ask it. I see there's a star there. What resources, if any, did you use to help your team get on board with using Git? Were you all pretty familiar with it already or did anyone on the team start learning from the ground up? I would say we've been learning from the ground up. And, you know, I was thinking about this recently that I must have taken 10 different times that like said, all right, I'm going to learn Git. And I go, you know, take some sort of online course and sort of learn the fundamentals. And it never stuck. Like, I just was like, I don't really get this too much. And it wasn't until actually we were having help from, you know, the folks in our technology team on how to set up our repos. What kind of sort of, you know, patterns do we use when we collaborate?

I just today was told we were using like a trunk based approach for when we do our branches and merges. So that was kind of news to me. But I would say like none of it really stuck until I was put in a position or I put myself in a position where like I had to use it. And then once I, I mean, once we started collaborating across the team and I, that's when I really saw the power of it because, you know, I'm from a place before Git, I mean, back in my day, it was like, you'd have some code and then you'd, if you needed to change it, you just were praying that it didn't break. And if it broke, you kind of were sitting there like thinking, oh boy, I hope I remember what I did so I can undo it. And it's like really liberating to not have that kind of be carrying around that mental luggage all the time about, about what have I changed over the last two hours? And, and could I, could I sort of recover what I had before? So, I mean, I'm a huge Git advocate right now. Like, there's a really good, good paper out there, Good Enough Practices for Scientific Computing. I think Jenny Bryant, who's a Posit employee is a co-author of that thing. And that's a really good, good, she also has a Git for R, I think, or something like that, or some book like that. Yeah, there's a lot of love for that resource in the chat right now too.

Happy Git with R. Happy Git with R. Yeah, that's the one. And that's like, you know, even nowadays, if I'm doing something and I'm not collaborating with anyone, and I'm in RStudio, I'm starting up a new project. I mean, I'm clicking that Git repo button every time because that's like, that's my safety net. So I'd say like, I don't have really, I mean, we were lucky we had people in our organization that helped us get over the hump. But, you know, the best way to get familiar with it is use it. So it's kind of a catch-22, I think. There.

it's like really liberating to not have that kind of be carrying around that mental luggage all the time about, about what have I changed over the last two hours? And, and could I, could I sort of recover what I had before? So, I mean, I'm a huge Git advocate right now.

Good question. What's your background view right now? Are you in New York right now at the office? Yes. So I work in Midtown Manhattan. So that's, I have a view of the East River out there. So, and so I can see from, if I go that way, I can see the 59th Street Bridge, and then over there I can see the United Nations building. Right. I was just in New York yesterday. I should have came by the office. Oh, yeah. Yeah. Did you see any of the lights? I did. Yeah, it was pretty cool. Nice. Nice. Yeah, I'm walking distance over to like Sachs and the tree and stuff. So the other day I walked over there. It's really cool. Cool. Shoot. I should have checked. I was right there.

Yeah. I got to meet with Russ, who's usually in the Hangouts too. Let's see. Let me jump over to, oh, Michael, you had a question in the chat. Do you want to jump in here? Sure, if you're talking about me. Hi, Michael Thomas. I also do a lot of data science work in the financial services space, particularly in risk management. So I was really excited to hear from you today, Matt. I appreciate it. I was wondering if you use Shiny apps at all to communicate results of like Monte Carlo simulations or stress testing and things like that and how that's gone, if at all, to presenting those types of results to non-technical folks.

Using Shiny: prototyping vs. production

Yeah. Yeah, we do use Shiny. One of the things that I get a little bit nervous about with Shiny is it's such an amazing tool. I would say that we've kind of limited our use of that to more of the model development piece. So when we're building a model and we want to communicate to the users or the model owners or the subject matter experts, the people who know this stuff, how this model works, if I put in this input, what kind of output, what kind of sensitivities am I seeing? Shiny has been a really great tool for that.

My concern about Shiny is if I build something that is super awesome, which is really accessible with Shiny, like you can build really amazing things, my concern would be that I don't view myself as somebody who writes production code. So if it were to become part of a core part of people's day-to-day lives, I'd get nervous about something along the lines of getting a call at 11 p.m. that the Shiny app's down and can you fix it, Matt? And I'm the only guy who knows how to fix it. So it's a bit of a double-edged sword. So we do try to really leverage our technology teams when it comes to building core functionality, software functionality that people need. So it's like with great power comes great responsibility type of a message I'm saying there. But we've had great success with Shiny. I think Shiny, it's a really great tool. But mostly in the prototyping and sort of early results stage, I would say.

Yeah. I hear that. No, thank you. Sorry. Do you hand it over to that team as an API? Is that the workflow you were talking about earlier? So yeah, we would sort of try to, if that model, we would either try to, I mean, we haven't really fully executed on this vision,

but we would be looking to deploy the model as an API and then allow, you know, kind of reduce the bar of accessibility to people there so that they could, if they wanted to perform some analysis, they could kind of do it themselves. Or the other path would be like, you know, try to sort of work with technology to build some sort of software that would be kind of in our stable of software, internally built software that people are using to get things done.

How data science work translates to customer value

Yeah. Hi. Matt, this is a super like naive question about the industry. And I'm curious about how the work that you do translates over to what like the analysts are taking to customers. And I'm assuming that there's something that differentiates like that's your value proposition for like our analysis is better. Right. Cause you've got competitors in the industry. And so I wonder if you can just give some context to like, like what makes it, what makes a report successful to you and to your customers and how do you like, how does that factor into like, how do you plan an analysis or how do you plan a model? You can come at that in any way that is useful. Sort of just wondering how, like, how does this data fit into the big picture of what your customers are looking for?

Sure. That's, I mean, that's not a, that's a good question. So, you know, the way our business works is we have a lot of different ratings teams. So all of those teams are sort of experts in their field. There's a whole bunch of different types of things that we're rating. So we rate corporates, we rate banks, and then there's this whole other side of the fence where it's like people issue debt that are, that's backed by sort of some sort of cash generating assets. So that's called structured finance. So like residential mortgage backed securities is a thing out there, commercial mortgage backed securities. So that, so each one of those things kind of has its team of subject matter experts that are in charge of like doing the credit analysis. They know all about what drives the credit, what could make the credit not pay, you know, they know all that stuff. So each one of those groups have one or more that we call the methodologies. So these are public statements to the world that describe everything that goes into, like, it's kind of like the playbook. It's not, it's not as sort of step one, two, three, like it's not like a recipe book, but if you're, if you know the industry, you can pick that thing up and you would have a sense of like, what is, how do we do it? How do we, what are we thinking about when we, when we, and I say we, our company KBRA does a credit rating. So, you know, the models we build are generally like a component of that, right? So we will build, you know, there's qualitative, quantitative aspects of the process and our work flows into the quantitative piece of that, but it's all, it's all part of a broader thing. So, you know, I would say that KBRA differentiates itself with like, we have, like I said, really strong credit rating experts who do the work and they do quite extensive write-ups on everything, every consideration they have when they're creating the rating. So it's, it's hopefully that helps illustrate a little bit of what's going on there.

Collaborating with Excel-first business users

Yeah, yeah, it does. Thank you. Yeah. Matt, I remember, I'm thinking back to our conversation at the POSIT conference and we were talking a bit about how you collaborate with business users or people that primarily use Excel company. And I was wondering if you could expand a little bit on that and share how you've been able to, I guess, effectively collaborate with business users.

Yeah. Well, we, we have, we have experimented a little bit with like writing sort of VBA code that we can share with people that would sort of allow them, give them access to, in a non-programmatic way, sort of to an API. Like if I were to give, if I were to publish the best API in the world, you know, I, I think there's probably 80 to 90% of the people who, who I work with just, it wouldn't be particularly helpful for them because they don't know that, they don't know how to interact with that. You know, there is quite a bit of, you know, it's always a, a decision to be made. I mean, we do, we, I've done some work where I, I've implemented my model in Excel and that's, that's okay too, you know.

So I think, you know, JD Long did the keynote speech at the POSIT conf last, last, I don't know when it was, recently. And, you know, he, he's a big advocate of like, you know, being mindful and empathetic for people because ultimately people, you know, they're just trying to do their job and they just want to get things done. And my job is to try to help them. So it's, it's definitely a challenge though, because what I, I do really want to sort of maintain some sort of, some sort of line between like my work and their work. And, you know, I don't really want to become their, you know, Excel support line type person. So that's kind of a thing I'm always trying to balance.

Large language models and security concerns

I see, Tim, you had a question a bit earlier about internal packages, but I see you might have a few. So feel free to pick your favorite question. Well, you already answered my question about internal packages and another thing you mentioned that you were writing some internal packages. Yeah. The question I just asked though, is are you, with a small team, are you using large language models like ChatGPT or things like that to assist with your code writing at all?

Yeah. So we are, I think that's in our future and that's something I'm really eager to, to sort of engage with and have that kind of assistance with. ChatGPT, GitHub Copilot is, I've sort of dabbled with like at home, but like, you know, we, as an organization, you know, I think there are some security concerns, you know, we don't generally like to, we, I mean, we are dealing with sometimes non-public information. So we are very kind of locked down. I work at a law firm, so I feel your pain. We have our own version of ChatGPT hosted in Azure that's just for us. So I mean, I will tell you, it is so much better at writing Roxygen comments for functions for a package than I ever could be. Oh, I can't wait. We have not, we have not yet engaged with that. That sounds like an awesome, awesome use case of that technology.

Yeah. But we haven't done that yet. And, you know, I sit right next to the head of security and, you know, I'm talking to technology and that. I do think that like at some point, everybody's going to have that tool because if you don't, then you're kind of just not performing as well as your competitors. Yeah. We're just not there yet.

I do think that like at some point, everybody's going to have that tool because if you don't, then you're kind of just not performing as well as your competitors.

Open source in the financial industry

I see Bill had a question a bit earlier that was, are you seeing more open source being used in the financial industry? I mean, we're using R and Python and pretty much our entire software, internal software development teams are using Python. So, yeah, we're in favor of open source. Yeah. I guess maybe it's also like, how have you seen it change over the past eight years of being there too?

Um, I mean, when I, so my company was formed in 2010 and I arrived in like 2015. So there's definitely like a startup mentality. So I would say like, uh, at that time, it, it was a little bit about just get the job done however you need to. I'd say that like we are getting more and more sort of, you know, as we grow, our control framework is, is maturing and we're getting more and more controls in place, but that hasn't prevented us from engaging with open source in any way.

Model deployment and consumption

Uh, Julia, I see you had a, another question around like what happens when you're done with a model? Do you want to jump in here? Yeah. You were talking earlier about how Shiny for you is mostly a piloting tool and also that you've been trying to put things behind, for example, API, like you put a very, very API up for a model. And my question is mostly what, what is it that actually consumes the models that you build? Do you have your IT build some infrastructure around that or is, will there be like a bat shop that gets run? Like how does your model then blub out?

I think I would say that, you know, we have not yet like standardized everything to the point where like we, I can say like we always do X, Y, or Z. So like, you know, it's been IT will, will sort of incorporate our models into their, into their software that they're building. Like I said, sometimes we kind of embed ourselves in the technology team and, and, and sort of put a hat on as a software developer to write the portion of the code that does that implements the model. Um, you know, sometimes, uh, you know, sometimes, uh, we've implemented our models with the worksheet functions and, and they're completely done in Excel. So there's no like real, what, what I'm trying to do is to sort of standardize things to the point. Like I would love to be in a position where if I were to deploy an API, then, uh, you know, everybody in the company would have access to it without a huge, huge hurdle to overcome.

I will say that, like, so, you know, I, I, I, I love the work that Max Kuhn and, uh, and Julia have been doing it with, uh, you know, the tidy model suite. And when, uh, Julia started talking about like vetiver and model deployment and model monitoring, it was, it was, it was really heartening to see that other people were thinking about that stuff as well. Cause that's like, you know, you could build the model and you could sort of, uh, and your job's not done. Your job's like way far from done. People, you have to figure out how to get people, how to allow people to use your model. And that's, that's a big, that's a big, uh, tough nut to crack.

So how do we get people to use it? Oh, well, so we're, like I said, we, sometimes we make it work in Excel. Sometimes we deploy an API and sometimes we become software engineers and start making commits to a Git repo. I guess I'm thinking too, like about the communication part of it. Like how, how do you work with them to help people see the value in some of the models or do they see that initially? Yeah, well, I mean, I would love to sort of like solve all the problems I currently have and put myself in a position to start like generating more internal business for myself by convincing people to trust their models more and, and, uh, and, uh, um, use, you know, use them more and more and more. But like, you know, we, we kind of are dealing with the problem in front of us right now, I would say.

Gaining trust and working with IT and regulatory approval

Yeah. Um, another question I saw just pop into the chat. Bill asked, in your organization, do you have to justify or submit for approval to an IT or regulatory group? Um, I mean, yeah, I mean, we have a, we have a, um, it's a whole, it's a whole thing, right? So we need to sort of describe the project we're making and then that sort of goes, we're, we're proposing and then that, and then that goes into sort of a prioritization decision-making type of a thing. And then, uh, and then, I mean, it's, it's, uh, not quite as formalized as other places I've seen. So it's not like I'm filling out like form, form T, T95 or anything like that, but there is a, I mean, we obviously, I mean, we're a business, right? So we, uh, we need to work on the things that are most important to us.

But, you know, um, I'm, I'm, I'm sort of lucky that I've been here for quite some time. I, you know, a lot of the, I'd say that early on while I was here, um, a lot of, a lot of what I was doing was, was work to sort of gain people's trust and that's a super important thing, right? So, um, and, um, not to pat myself on the back too hard, but I did it, right? So people, I have worked hard to make sure that people can trust me. So like, you know, uh, this most, we just recently kicked off a new project where it's more of that software development type of a, of a lane that we're getting into. And that, that one for a long time, I was really sort of trying to figure out

how do I do this job? I don't want to be, I'm not a software developer, but I kind of play one on TV sometimes. Um, how do I do this job where I, you know, where I can keep a low profile and do things autonomously. Um, and I kind of concluded that, that there are certain things that you do need to, to get in the queue, get in that prioritization sort of, uh, game. Um, and, uh, and that's the best, that's going to be the best outcome because, you know, you don't really want to be creating in an organization like a shadow IT or organization or anything like that. I think that's, that it's great for getting things done, but I think down the line, you know, we pay a lot of people who really know a lot about technology to do that job. Like, I don't want to have, you know, another one of those run by the handful of people I have, right, that become really important. Right. So I'd rather be on their, on their turf.

Opening it up to the group

Are there questions that you'd like to ask the group here or things that you'd love to maybe open up to more of a, a discussion on and trying to think back to what we were talking about at, at the conference as well. And we were, um, I hadn't really contemplated that. I thought you guys were going to ask me questions. No, I cannot keep going on this. I just wanted to make sure that like, if there are certain problems that you're facing that you're just curious to open up to ask everybody here that we, we give the time for that as well. Yeah. Um, no, if somebody has any good ideas after hearing like what we're dealing with, I mean, you know, a lot of the, a lot of the sort of difficulties sort of we've encountered being in this role between like the credit rating analysts who are doing their job, they want to rate the, they want to do the credit analysis and technology who's also doing their job is that, you know, it really opened us up to it being just people not really understanding what our role is. Right. So like we, lots of times we really do get confused with technology. Like we'll get a call from, I'll get a call from senior leadership, people who are very senior in the company and say like, I need this thing that does X, Y, and Z. And they're not describing a model. They're not describing like really anything I, I, I'm, I do, or I'm really good at.

Uh, so, you know, that sort of establishing, like trying to continually hammer home, like what described to people, what is it that I, what would you say you do here? You know, I, I, that's a, that's something that I, I, um, I've sort of encountered a lot like that, that clarity and making sure that people understand that. And then just that idea of like trying to really, because we're, we're a smaller team and, you know, we do have limited resources, um, just establishing those lines of ownership, right. And helping people without like, you know, doing their job is kind of a thing that, that can be difficult too. So those are things that kind of need to be managed, I would say. And that's what a lot of, I wouldn't say I spent a lot of time on it, but I've spent a lot of time thinking about it in the past. And a lot of the, like a lot of the things we've done around, um, you know, using the right tools to, to get our, our work repeatable and this type of thing. And being able to engage with technology sort of on their turf in certain instances that has, has really been beneficial.

Audience Q&A

I see a lot of great questions, uh, jumping into the, or coming into the chat here. Um, Laura, do you want to ask your question here? Sure. I'm going to stay off camera because I'm a little under the weather. Um, but, uh, so my question is, have there ever been situations where rating against maybe a portfolio or a company has come out against what the gut check is? I'm always interested in those situations where an analysis goes against the initial hypothesis and how organizations handle that. So I'm wondering if you have some situations and how you've navigated those. Yeah. I mean, I would say that like, you know, we, I mean, none of our ratings are like the model spits out the rating and that's what we go with. Right. So it's always, there's always a level of human oversight and human intervention there. I would say, I mean, obviously with any model, you're going to have some outcome that somebody who really knows the domain is going to say, no, I don't think that's the deal. I don't think that's what it is, but you know, our methodologies sort of really accommodate for that, right? Like these are, there's always a qualitative and a quantitative component. And, and ultimately like the way the process works, there's a rating committee and that the decision is made not by my model, right? My model's an input into that process.

You know, part of the reason why we, you know, earlier I said, we, we, we rely, you know, given the choice between, you know, a model whose accuracy ratio is 1% greater than this other model that like I can explain to somebody how this model got this answer. Like I'm, I'm pretty much always going to go with the explainability part. And your question is kind of like, is like part of the reason for that is like, if I get a question, why did the model give me this answer? Like I, I'm, I'm very happy to explain to them. And, and then that could sort of drive a conversation about like the limitations of the model. Maybe that would be an indication that, you know, maybe it's time, maybe this model has exceeded its use, utility, and we need to build and re-estimate it, build a new one that has other, other risk drivers in it, you know, those types of things. So it definitely happens. We're never really put in a position where something goes out the door that sort of came out of a model and nobody looked at.

R.M., I see you have a question in the chat as well, if you want to ask this next. Yeah. Hi, thanks. Small question. If I understand your organization correctly, I'm trying to understand whether you consider yourself to be a embedded within the business units or more of a centralized operation for your data science work? Yeah, we're a centralized sort of function, a support function, really. And then more on a project type basis would we consider ourselves like embedded with a team. So we'll, we'll you know, or if we're working on a project and we need to have the subject matter experts in the room, like then we'll sort of in that way. So on a project basis, maybe we would become embedded, but we're never truly embedded because, you know, we don't run the models and we're, we're not really engaged in that rating process, like on the ground so much.

I saw one thing, always qualitative, quantitative and qualitative part of the model. I, maybe I misspoke. I meant like the, the rating, the rating process or the rating, the analysis is what I maybe should have said. I think we have time for a few more questions here. But Alan, I see you had another question in the chat. Want to jump in? Yeah, sure. Thanks. I, I'm curious, Matt, you, you mentioned at one point having things that sort of you want to do, you want to develop sort of products that you want to develop. And I'm curious what the balance is of things like that, where you get to sort of be entrepreneurial versus, and that goes into some kind of portfolio of prioritization that I think a lot of us are kind of familiar with conceptually versus things that kind of come into your team from the businesses or from other customers. How do you balance those things? What kind of help do you have in, in picking, you know, when to do one versus the other?

Yeah. I mean, this is getting back to that idea of like we employ some of the tools that are available to technology. So like we have a, you know, a support mailbox, that's the quantitative modeling support mailbox. So if somebody has a question or something that they want us to work on, you know, they send a, they send an email there, it automatically creates a JIRA ticket and we manage, we use JIRA to sort of manage our work. We do maintain like workflow limits. So we don't really want people working on more than two or three things at a time. And so far it's worked. We really haven't had this, that sort of like a moment of like, oh, do I do this really important thing or that really important thing? We've kind of been able to use that to help us not only like not forget to do things, which is a huge, huge benefit, but like manage our workload. But like, it also gets back to that, like idea of like building trust. So I've been lucky that, you know, I've, I've had the opportunity to sort of do things that are sort of have that overlap between stuff that my company needs to get done and stuff that I'm interested in doing. So that's been really good. And it's been nice to sort of be able to, to sort of call the shots along the way.

Jeffrey just put a question into the chat. How does your team work together? Do you have a data engineer that pulls data, passes it to a data scientist, or does that, the person that creates the model do it all or something else? Yeah. I mean, it's really is on a project by project basis that sort of the genesis of what that data set looks like. It sometimes, sometimes the team who's responsible for the rating have some, have access to like really great data sets that we can use to build our models. Sometimes it's about like pulling it together from like public, publicly available sources, like the, you know, Federal Reserve Economic Database or other types of things and pulling that together. I will say that, like, I did neglect to say, we do have embedded on our team, like folks who are like that MLOps type role. So they're more technologists who are helping us like maintain our platform. And also like they're Crackerjack coders who could sort of step in and give us some feedback on how to do things, how to get, how to make Git work better. And also how to like engage with some of the internal resources and technology. So we do have some folks on the team who are more of that technology role who are helping us get things done. And sometimes they'll be, they'll be useful in that sort of data, more the upfront data acquisition.

Building trust with stakeholders

I know we talked a little bit about trust or came up a few times and I was just wondering, what are some of those ways you were able to build trust with the stakeholders? Yeah, I'd say like early on, and it was, I used the Shiny app to get it done, right? So early on, this was before when I was doing like the model validation stuff. So I kind of had, I was starting to get insights into like, what are the sensitivities? What are driving this model? How does this thing actually work? And I had just learned Shiny. So I was like, I took that knowledge and I sort of built a Shiny app that allowed people to like move a slider, press the button, see the graph type of stuff. And I would say that like people's socks were knocked off. They were blown away. So those types of moments are like, not just useful because now they have something that they can look at the sensitivity, but like from a more personal perspective, like suddenly people were saying like, oh, Matt kind of seems to know how to get things done. Maybe we should trust him. So I think it's like looking for those types of opportunities. And it's never like a, there's no playbook there. It's just, you got to kind of keep your head up and you got to know what you're capable of and deliver. And that's really useful.

It's just, you got to kind of keep your head up and you got to know what you're capable of and deliver. And that's really useful.

Love it. That's great. If people want to get connected with you, Matt, is the best place LinkedIn? Yeah, LinkedIn is good. LinkedIn is good.

Awesome. Well, I did just want to remind everybody that we did, I think this was probably five weeks ago now, but we had a lengthy conversation on how to estimate the ROI for a data project. And there was so many thoughts on this and conversations in the chat that we decided we wanted to have a separate follow-up conversation on that. And so I shared that in the chat a bit earlier. If anybody wants to add it to their calendar, let me try and copy it over again. But that's going to be on December 13th. So it's on a Wednesday and it's just like a bonus data science hangout chat for us. Let me see. Here it is. Copy my message from above and put it there, but I'd love to have you all join us on that. If it's something that was of interest to you when we talked about it before, feel free to add it to your calendar. But thank you so much, Matt, for joining us. Thank you for hosting. I really enjoyed it. Thank you so much. Yeah, thank you. Well, have a great rest of the day. We actually think only have one more data science hangout for 2023. So that will be next week before we take our holiday break for the hangout. So hope to see you all back next week. Thanks, everybody. Bye. Bye.