image: thumbnail.jpg
Model Monitors and Alerting at Scale with RStudio Connect | Adam Austin, Socure
A monitor should provide insights into model inputs and output scores, and should send alerts when something goes wrong. However, as the number of deployed models increases or your customer base grows, the maintenance of your monitor portfolio becomes daunting. In this talk we’ll explore a solution for orchestrating monitor deployment and maintenance using RStudio Connect. I will show how applications of R Markdown, Shiny, and Plumber can unburden data scientists of time-consuming report upkeep by empowering end-users to deploy, update, and own their monitors.
Timestamps:
2:01 - Start of presentation
3:43 - About Socure
6:12 - Model performance matters, deployment isn't the end of the story
7:26 - What is a monitor?
8:55 - What is an alert?
11:00 - Monitor example
18:09 - Firing an alert from RStudio Connect
19:00 - Why monitor from RStudio Connect?
24:33 - How monitoring drives success at Socure
30:00 - Git-backed deployment in RStudio Connect
36:00 - Shiny app that their account managers see
46:00 - Architecture of a monitoring system
56:00 - Connect hot tip System-wide packages
57:00 - Why did we try Connect for monitoring
59:02 - Why do we keep using it for that :)
Resources shared:
Blastula package: https://github.com/rstudio/blastula
connectapi package: https://github.com/rstudio/connectapi
rsconnect package: https://rstudio.github.io/rsconnect/
Intro to APIs blog post: https://www.rstudio.com/blog/creating-apis-for-data-science-with-plumber/
Speaker Bio:
Adam Austin is a senior data scientist and RStudio administrator at Socure, a leading provider of identity verification and fraud prevention services. His work focuses on data science enablement through tools, automation, and reporting.
Transcript#
This transcript was generated automatically and may contain errors.
Thank you so much for joining us today. Welcome to the RStudio Enterprise Community Meetup.
I'm Rachel. I'm actually calling in from Connecticut today. We are streaming out to LinkedIn and YouTube Live right now. Well, hopefully we're streaming out to both right now. But if you just joined now, feel free to say hi through the chat window and maybe where you're calling in from. Today we are joined by my friend Adam Austin at Socure, who's going to be sharing how they monitor models and alert at scale with RStudio Connect. If this is your first time joining one of these meetup sessions, welcome. This is a friendly and open meetup environment for teams to share use cases, teach lessons learned, and just meet each other and ask questions. Together, we are all dedicated to making this an inclusive and open environment for everyone, no matter your experience, industry, or background.
I'll share a link in the chat in just a moment here where you can find out about other upcoming events too. During the event, you are able to ask anonymous questions as well, in addition to writing in on LinkedIn or YouTube, wherever you're watching from. And we'll share that link to the Slido in a second here too. We'll try to answer as many questions as possible after the presentation too. But with all that said, thank you so much for joining us here today. And thank you so much, Adam, for sharing your experience with us. I'll pull Adam here in a second, up here in a second. Adam is a senior data scientist and was previously an RStudio administrator at Socure, a leading provider of identity verification and fraud prevention services. His work focuses on data science enablement through tools, automation, and reporting. Thank you so much, Adam. I'll pull you up here on stage with me.
How are you doing today? Doing well. How are you? Great. Thank you for joining us. I'm going to head backstage and turn it over to you. All right. Thank you so much and welcome everybody. Thanks so much for joining today. Like Rachel said, our topic today is model monitors and alerting at scale with RStudio Connect. And I just wanted to give a shout out. I believe in the audience are Gordon Shotwell, who is a colleague of mine at Socure, and Jonathan Keene, who is a former Socurean, as we call them. And they were both instrumental in sort of developing the foundation of this whole monitoring and alerting framework. So shout out to them. I'm sure they'll keep me honest in the chat throughout the call because they were kind of there in the beginning. And I'm just a messenger in this whole voyage.
Oh, I lost your audio for just a second here. You're right. Sorry. That was my. Okay. Okay. Cool. Thank you. Yeah. So a bit about me, I've been doing data science for about eight years. I've been at Socure for about a year and a half. I'm sort of Chicago adjacent. So I'm out there in the Chicago suburbs. And if you want to follow me on Twitter or like go find my GitHub, you're welcome to do that. I don't have a lot of time to tweet or to develop, unfortunately, because I have a whole lot of side projects. So this is it right here. This is my fam. And yeah, anyway, that's a little bit about me.
About Socure
So I do want to talk about Socure briefly, just to give you a little bit of context for where I'm coming from. And if you did see Gordon's talk, Gordon Shotwell, my colleague, he spoke at the meetup a few months ago, I believe, about security. And he gave a little bit of a background about Socure, in case you weren't there. So here is, it's a company that provides identity verification and fraud prediction models. And so we're essentially a business to business provider of model scores and insights via our API. So we're not consumer facing. So if you haven't heard of us, that's just because we don't really sell to the general public, we sell to businesses. And the idea there is there's a number of firms like banks, is a good example, have regulatory requirements around who they can do business with, and around a certain minimum standard of authentication verification of their users. And so we provide services around that to them.
Also things like financial institutions, and any firm that has an interest in moving money around financial transactions, or essentially working with your private personal data, might also want to prevent fraud, because it's just good for business. So they want to keep losses low. So those are our clients. And to that end, we deploy multiple kinds of models. And we serve model scores to our clients through our APIs. And our clients come from all kinds of industries. So when I say we deploy a model, what I mean is we develop a predictive model based on all kinds of attributes about a person. And those models basically say, are you who you say you are? And do you have good intentions? So that's kind of what our business is. And so we will put a model in production, which means it's behind an API that a business can then make a call to and say, here's somebody applying for an account at our institution. Can you tell us, is this person legit? And do they have good intentions?
Why model monitoring matters
So with that out of the way, why are we here today? So we deploy models, like I said, but that really isn't the end of the story. The performance of that model matters, obviously. It matters over time. It's going to matter at the model level. And it's also going to matter at the customer level. So we want to make sure that our models over time are producing reliable, stable scores. And we also want to make sure that each individual customer is receiving the kinds of scores that they expect and that they can make use of.
So what we'll cover today, well, the name of this talk is model monitoring and alerting at scale with RStudio Connect. So I thought we would cover, first of all, model monitoring and alerting, how we scale it up, and how we do that specifically in the RStudio Connect framework. So I think of this as the what we do, who, when, and why you might want to scale it for, and then where it's all done.
What is a monitor?
All right. So first of all, model monitoring and alerting, what does that even mean? So what is a monitor? In its simplest terms, to me, a monitor is a visual presentation of model metrics that is refreshed regularly and will notify you if something is wrong. This is sort of the three components I think of when I think of monitoring. So basically, making sure that everything is running smoothly. And it's a place you can go to visually see what's happening. It's something that should be constantly updating, so it's always kept current, so you're never caught off guard by anything that goes wrong. And it should alert you if things do go wrong.
So I use that word alert. What actually is an alert? So this is going to be some kind of notification of abnormal data inputs or perhaps model outputs. And you want to be alerted if something goes wrong so that you can respond quickly. Maybe you need to retrain your models, or maybe you need to look into a problem with an upstream data source, or maybe you just need to communicate with a customer. Or in our case, in Socure's case, we want to communicate with the customer and say, hey, we noticed that our scores are changing. Is something different on your end, or should we do some more digging on our end?
So an example of an alert might be, you know, suddenly you're getting too few records from an input data source. Maybe you have just sudden spike in bad input data, so too many missing values. Maybe you're getting the wrong factor levels coming through and your model can't score those. Maybe you have certain input values that are exceeding expected thresholds, so they're too high, they're too low, and that just looks weird to you. And also, very importantly, you want to make sure that your models are producing the right kinds of scores. So when you're building a model and vetting it, you're going to produce a sort of certain expected distribution of model scores, and you want to make sure that those things don't change over time through, you know, drift in the kinds of records you're scoring and that sort of thing. So if your model scores change considerably, that might have impacts in Socure's case for our clients in terms of how they're thinking about their customers, so you want to make sure that we alert them to that.
Using R Markdown for monitoring
All right, so how are we actually going to do this monitoring and alerting? Well, this is a great job for R Markdown, or of course, Quarto is sort of the new R Markdown, if you will. It's a bit of a replacement for R Markdown, but when we were developing this system, we've been using R Markdown, and it works really well for us. So why would we use R Markdown in this case? Well, it has, obviously, infinite flexibility. You can compute any function of your data.
And this is important because there are a lot of really great monitoring solutions out there. Different kinds of companies provide ways for you to monitor your data and your models. A lot of those solutions are a little bit limited in how you actually compute functions of your data. So maybe you're forced into using certain kinds of SQL queries, or maybe you're forced into using a kind of certain proprietary language to compute and construct your statistics, and that can be a little bit limiting. With R Markdown, you're just using R or even Python if you want, and so you can compute any function of your data.
R Markdown is great for hosting because you can publish and refresh your reports on RStudio Connect, and we'll see more about that in just a minute. And R Markdown is really nice because R has a huge suite of integrations. You can choose to send out alerts via email. You can use Slack. You can use GChat. You can use Microsoft Teams, and so on and so forth. There's all kinds of great integrations where you can choose different channels to send your alerts to based on your needs. I think email is nice because even though it requires more setup to get going, I think there's a bit more flexibility in what you can report to end users there, and it also allows your users who are receiving these alerts to use advanced filtering and things like that to meet their own needs.
Anatomy of a monitor
Okay, so just to kind of level set, what's an example of a monitor? Again, it's just R Markdown here at Socure. We have a nice custom template that we apply to R Markdown so that when it renders, it has this cool theming. And again, it's just going to be a report, and it's going to visually present to you some information. So, maybe you're interested in, again, the number of input records you have over time, the percent of missing values you get over time, so on and so forth. So, you can go to this report. You can examine all these things over time. And then also, once this is deployed to Connect, this can refresh and send you an alert if there is a problem. So, you don't have to be constantly watching this thing all the time because as you scale up and you have more and more monitors, that just becomes too time consuming, too burdensome.
So, let's discuss what the anatomy of this thing really is. So, this is sort of a nice polished, finished product. But what's actually the guts of this thing? What's going on? How does this thing come together? I think of this as having three parts. There's going to be a header section. There's going to be sort of the bulk of the monitor, which is around making sure you can get your data, you transform that data, and then you can visualize it, create plots and things like that, tables. And then there's a final component, which is taking that output data and deciding whether or not you're going to alert. So, we'll walk through each of these sections and see what these look like.
So, first of all, the header, if you've written an R Markdown file before, you've certainly seen this YAML header before. So, that's where you specify your title, your author, the kind of output you're going to create. So, if it's HTML, PDF, et cetera. But I do want to call out one particular thing that you can add here, and this is especially for RStudio Connect. If you're going to be alerting from Connect, you want to think about when those alerts are going to fire. So, when you deploy a report to Connect, you can choose to set that report to refresh on a regular basis. And when it refreshes, it's going to want to send an email. What you can do is you can actually suppress that email until you absolutely need to send it, and that's going to be when you have an alert. So, there's this little block in the YAML header you can add, the RMD output metadata section, where you can say, I want to generally suppress an RStudio Connect email, and then I'll turn this off later on if I have to fire an alert. I'll show you that in a second.
Okay. In the middle part of this monitor, this is just where you're going to have all of your logic, all of your R code, maybe your Python code to pull your data, compute your statistics, plot things, create tables, et cetera. This is all just what you would expect from a standard R Markdown file. Now, the alerting part is kind of where some of the magic happens. So, again, it's just part of the standard R Markdown workflow. This is just another R code chunk in this case, but we're going to be relying on two packages in particular. One is obviously the R Markdown package, and one is the blastula package, and blastula is this really
cool package to help you email out reports. It makes it super easy to do, especially in the RStudio Connect environment. So, this is just kind of a high-level overview. Here, I'm taking my previously created statistics and functions of my data, and I'm going to process some alerts, and at this point, I'm going to say, do I have any alerts? That is, do I have anything that I see concerning in my data? Do I have too many missing values? Do I have model scores that are too high or too low? Whatever it is, and then you can decide in this code chunk whether or not you want to fire an alert. So, if you do have any alerts, at that point, you're going to use the R Markdown package to then say, I'm no longer going to be suppressing my emails.
I'm going to tell Connect, I'm ready to send an email now, and then next, I'm going to use the blastula package to render my Connect email, and I'm going to use this alert email template, which I'll show you in a second, and I'm going to attach this email, and I'm going to send it off to my recipients.
So, this does require a configuration on the RStudio Connect server, so if this is something that you want to do, you can talk to your RStudio administrator, and they can help you get set up with these emails. Okay, so I mentioned this alert email template, and let me just show you what I mean by that. So, you have here, you have a monitor markdown file, and then what you're doing is you're calling a secondary markdown. In this case, it's just going to be an alert email template, and I call this the alerting buddy system. So, if you've deployed this monitor to Connect, you're also going to want to deploy this little helper markdown, and this is just going to be, this is going to consist of a header and a body, and the header now is going to contain this blastula function, which specifically renders this as a blastula email, and then in the body of this alert markdown, it's just going to be any content that you want to report. If you want to put a table in there, if you want to just put some custom text that just says, hey, there's an alert, go check your monitor, you can do that. If you want to put a table, a plot, whatever it is, you can put that all in here.
So, one thing to be aware of here is that this alert email markdown file, when it's called in your code, it's being rendered within the scope of your original monitor. So, you don't need to recompute anything here. You don't need to pull new data. All of the objects that are available in this environment here in your monitor are going to be available in your alert email as well. So, that's just one thing to keep in mind.
Alert email example
So, what does that actually look like when it fires an alert? In this case, we have Gringotts Bank, and maybe they had some model score drift, and maybe they had some input variables that have exceeded their thresholds. And so, in this case, this alert goes out and comes into my inbox overnight because that's when the report ran. And this just gives me a little bit of information. So, this text was rendered in my alert email R markdown. And then what's nice about this is this comes into my inbox. I get in first thing in the morning, I see there was an alert, and now I can go click down here, and I can just go right to the report and see those plots and make sure that everything is looking good. And if it's not, I can go investigate.
Why monitor from RStudio Connect?
Okay. So, that's what a monitor is, sort of how the email system works here. Why would we want to monitor from RStudio Connect in general? Well, it's unreasonably easy to deploy reports there. It's super nice to be able to tag and categorize your content. And so, what those tags do is it allows you to search and filter and find your monitors, find any reports or content that you deployed. You can also really easily schedule your refreshes there. So, you can schedule a report to run overnight, you can schedule it to run every hour if you want more frequent updates. And there are tools for scaling. And that is the crux of what I want to get into next.
But first, I'll stop and just see if we have any questions that came in from that first section. Rachel, anything you want to call out?
Let's see. I saw one question over on LinkedIn, but I first wanted to remind people that you can ask questions wherever you're watching from. So, LinkedIn or YouTube, but you can also ask questions anonymously at this Slido link too. But one of the questions that came in was from Gary, and it was, does this tie into site reliability engineering principles and practices?
That's a great question. I mean, in so far as you need to monitor and evaluate processes that are happening. So, I mean, so this, I guess what I called out here is specifically around, so I called out here specifically like a data process. So, do I have too few records from a data source, bad input data, et cetera. You can theoretically use this to monitor anything you want, right? So, anything that's going on in your organization, as long as you have the data, yeah, you can absolutely use this as a monitor for those processes. I think the challenge there is making sure that you have access to the right data sources and are able to hook that up to your Connect system, but otherwise, yeah, absolutely. You can keep tabs on anything you need in terms of who's accessing what systems and the uptime of your systems, as long as you can get that data.
Awesome. Thank you. I see one other question that came through on Slido that we'll cover now too, and it was how much tweaking is required to make sure you alert when necessary and don't when not? Do you have a standard process for that tweaking? Yeah, absolutely. That's a really great question, and we're going to address that in the next section. Yeah, so, because you can think about as you begin to scale up and deploy a lot of monitors, you might run into the situation where you're alerting a lot, and maybe you've set your thresholds to the wrong levels, and you absolutely want to keep that in mind as you begin to grow the number of monitors that you're deploying, and we'll touch on that in the next section. Perfect. Thank you. Cool.
Scaling monitors
All right. All right. Next section, at scale. So, it's one thing to build a model monitor and make it fire alerts. You know, it takes a little bit of setting up, but once you've got it going, it's super nice. Then, how do you scale this up? And so, I want to talk a little bit about our journey at Socure and thinking about when do you need to start scaling, and my Mil2 boys are just, like, super into Power Rangers right now, so this just came to mind thinking about scaling.
So, you want to start scaling when you have a sudden growth in the number of models that you're deploying or maybe clients who are receiving your reports. Maybe you have a large customer base either inside your own company or outside your company, and you just need to be able to keep tabs on a lot of different things at once, and again, you can't spend all day just staring at monitors, so you want to make sure you have a system by which if you have a lot of things going on, you can only get those alerts that really matter. You also might consider scaling up when you're overwhelmed by making updates, you know, to that question that we just had. You know, if you have tuning, if you're thinking about how do you tune those thresholds and how do you alert only when you have to, maybe you want to make that more of a self-service process for the users of your monitors, and that would be a good opportunity to think about how to scale.
so you want to make sure you have a system by which if you have a lot of things going on, you can only get those alerts that really matter.
So, I'm going to walk you through how this process happened at Socure, and maybe your business doesn't look quite like this, but hopefully you can at least learn from our process and find places where the same ideas and principles can apply to you, or at least maybe learn from, you know, just some of the growing pains that we had as we were going through this as well. So, some of the background here is that Socure grew very quickly in the past couple of years, and our revenue doubled year over year for a few years in a row, and so we were growing our customer base significantly, and that became kind of challenging going from just a few monitors to all of a sudden we need this whole ecosystem. So, I'll just kind of walk you through that process, and while I'm doing that, again, I'm sure Gordon and Jonathan can keep me honest because they were there in the beginning when all this was started, so hopefully they can call out some added info as necessary.
But let's first look at how monitoring helps Socure succeed, and to do this, I want to just kind of simplify our business model a little bit just for the sake of illustration. So, you know, in simple terms at Socure, we have a data science team, we have some account managers, and they're responsible for understanding client needs, communicating some of that back to our data scientists, and sort of being, you know, being these liaisons and reps for Socure, and then what happens is, like I mentioned before, our clients will call our API, and they will get model scores back as a result, and then all of that data lands in our databases so that we have visibility into what's actually happening with our products, and from there, once that data is in our databases, our data scientists can publish a monitor or a report that ingests that data and displays, you know, some of the stats, some of the health of our business. So, in broad terms, that's kind of how we think about it, and that's the framework we're going to use for thinking about scaling going forward here. So, once those monitors are deployed, our account managers can view these reports, these monitors might fire alerts back to our account managers, and then based on that information, our account managers can decide whether and how to communicate back to our clients what's happening with our products.
Level one: bespoke monitors
Okay. So, let's think about the scaling process and sort of the level one. The first step you might take in monitoring a client or monitoring a model is just to create something bespoke or just to create a custom monitor for every individual client or every individual need. When you create a custom monitor for every individual client or every individual need, when you're small, it kind of makes sense to start out that way because, you know, maybe you only have a few customers or a few models, and you're kind of learning which metrics are useful and what's the right refresh cadence and how do we send alerts and so forth. So, perhaps it looks like this where you get a new client and the account manager comes and says, we need to monitor Cocoa inputs and the Bad Egg model scores for this new client Wonka, Willie Wonka. He's our customer now. So, data science says, okay, cool, we can do that. We're going to build a custom monitor. We're going to publish that to Connect, and now our account managers can view that and can receive alerts.
Okay. Time goes on, and now we acquire some new customers, and now we have Gringotts Bank, and now we also have Dunder Mifflin. These are our new clients. So, our account managers say, okay, we're going to, you know, look at some specific model outputs for these clients, and so data science goes and publishes custom reports for those clients as well.
So, again, this is still sort of the prototyping phase, figuring out, you know, which metrics are actually useful, and this is a good place to start. The trouble here is that this can be very time-consuming. Maybe every monitor looks different, so, you know, it's a little bit hard to go from one monitor to the other and understand sort of what's happening in each one. There's a little bit of, like, cognitive overhead there.
Level two: standardizing via template
So, as time goes on, you will want to begin to standardize via template. Sorry, this should be level two. So, in level two, you're going to be thinking about, can we monitor some key inputs and model scores across all the clients? So, maybe there are some clients that require a little bit of extra customization, and that's totally fine, but can we figure out maybe a templated solution where, for most of our clients, we just know these are the inputs we care about, and these are the outputs we care about? So, customization kind of becomes secondary, and you just want that consistency, those standardized reporting practices.
So, what might that look like? Data Science might decide to create a report template, and this is how we did it for a while as well, and this template is going to contain a lot of logic in the body of that report, and all that logic is going to be essentially the same for each client, but what you'll do is you can go in in each report and say, you know, this report is for Willy Wonka, or this report is for Gringotts, or whoever it is, and then the downstream logic can then go and grab the data specific to those clients, and then each report can be
a templated version of this standard monitoring framework, except it's client-specific.
So, in our case at Socure, what we did is we created a template, and then we customized each template and deployed into a GitLab repo, and so now we have a whole bunch of monitors, each one showing the same kinds of things, but each one for a different client, and then we can use what's called this Git-backed deployment process in RStudio Connect to actually pull those reports straight into Connect, which is super nice, because if you wanted to go in later and update one of these templates, maybe you wanted to tweak a threshold value for one of your alerts, you could do that, you could push those changes to GitLab, and then maybe the next day RStudio Connect is going to go grab those changes and then publish those reports on Connect.
So, this was really helpful. This is a really nice system for a while. The problem here is that this is a little bit time-consuming, and there's a multi-step deployment process that can be a little bit cumbersome, again, as you continue to scale up, which is what happened to us. So, time goes by, and all of a sudden we have a lot more clients, and our account managers are asking for a lot more monitors, and so we have to begin, again, thinking about how do we reduce a little bit of the friction on the data science team, and how do we make this, again, even still more automated.
And to that end, we came up with a package, and this package was all about deploying a report by combining or marrying this template and now a client-specific config or a configuration, and those two things together create a monitor.
The auto-monitor package
So, broadly speaking, in this auto-monitor package, and I'll talk more about these specifics later, but broadly speaking, there's this deploy function, and deploy just says, give me your account name and give me your configuration, and I will put together a template and a configuration, and I will create a monitor, and I will deploy that. So, this template can live inside your package, so you just throw it in the int directory, and then this configuration is just going to be a list of client-specific details, monitoring thresholds, and so forth.
Also, inside this auto-monitor package are just your standard monitoring functions, and so those are what get used inside your template, and you can use – and this is, like, super cool – you can use the rsconnect package and the connectapi package to add viewer permissions. You can set your refresh schedule. You can tag deployed content so it's easy to find. You can do all these really cool things programmatically, so you don't have to deploy a report and then go to Connect and configure everything manually. You can do all of this programmatically from R.
you can use the rsconnect package and the connectapi package to add viewer permissions. You can set your refresh schedule. You can tag deployed content so it's easy to find. You can do all these really cool things programmatically, so you don't have to deploy a report and then go to Connect and configure everything manually. You can do all of this programmatically from R.
So, now, with auto-monitor, we can just say, I want to deploy. I'm going to deploy for, you know, this specific client. Maybe you have a set of config defaults, so you don't have to worry about that either. You can just programmatically run through all of these client names and then push out all these reports from your console.
So, now, you're cooking with gas. Everything feels good, except the problem here becomes you have all these reports now, and unless you take the time to go in and customize all of the configs and all of the alerting thresholds – and this gets back to our earlier question – what's going to happen? You have these default set of thresholds, and because they're not client specific and customized for the client needs, you're just going to get tons of alerts, right? So, you kind of want to set these overly cautious alerting thresholds so you're not missing anything, but because each client has different needs based on their business, you're probably just going to end up with a mess of alerts, and this becomes overwhelming. You get a lot of false positives. You get a lot of alert fatigue, which just means people are just going to stop reading their alerts because they're just tired of staring at them, and what ends up happening is you're going to require a lot of updates to these monitors by hand by your data scientists, so you end up kind of in this Jira inferno, right, where account managers need updates to their monitors, and so they're submitting all these tickets, and they're saying, please update these thresholds so we're not getting all these alerts because we're alerting too frequently, and this threshold is not relevant to my client, and that becomes overwhelming, and data science makes the rage face.
A self-service monitoring ecosystem
So, time goes by, and what we came up with is a self-service monitoring ecosystem, and data science says download Jira tickets, long live self-service, and now here's the old world where account managers are getting all these alerts. They could submit tickets, or in our case, what we chose to do is deploy a Shiny app, so now this is a web app that faces our account managers, and our account managers can go in, and in that Shiny app, choose their clients, and then change the configurations on those configs, so they're changing the thresholds at which alerts are fired, and that app will then go and update those monitors so that you're actually getting fewer alerts, and what you're getting is not just fewer alerts, you're getting the right alerts, so you're getting the alerts only when they matter.
This was a little bit hand-wavy, and there's actually a lot more going on behind the scenes, and I'll take a look at that in just a minute, but first, I want to give you a little bit of a live demo of what that Shiny app looks like. This is actually an example of what our account managers see, and it's basically a list of all of our clients. It gives the account ID, and then it tells you what was the last alert date, so the reason we're showing this here, and we actually sort by this date, is if you wake up in the morning, and you get all these alerts, and you're just sick and tired of it, you can go to this app, and you can find the accounts that are alerting right away, so let's update Gringotts, so I'll click on Gringotts here, and what pops up is a panel where I can configure my – well, I can configure my config file, so what I'll do is I'll open this general settings tab, and now you can see that this is all of the parameters and things that go into our alerting framework, so let's say we just deployed a new model for Gringotts, so I want to go here to my model names, and I want to add this new industry-specific model for Gringotts, and I want to get rid of this old fraud model. Okay, do that, and now I notice that my monitor is only showing me the past 90 days of history. I'm actually interested in 180 days of history for that client, and with that, I will say, okay, I'm ready to go. I'm going to hit update. Okay, so I've updated the config for that client, so the next time that report refreshes, it's going to use those settings, so I can go back to Gringotts, and I can see indeed I have my industry-specific model here, and I have my 180 days of history.
All right, so again, this is still high-level, kind of hand-wavy. I'm going to show you in a bit exactly how that gets executed behind the scenes, but that's the end of this section, so that's the idea of how we scale. Let's look here, and let's pause for questions if we have any.
Q&A
Yes, we do have a few questions, and I also wanted to bring Tatsu on, because he was going to be my co-host, and I completely forgot to send him the link, so welcome, Tatsu. Thank you for joining us. If you want to also introduce yourself to everyone.
Hi, everybody. Go ahead. I was just going to say, so Tatsu works with many of our consulting and high-tech companies like Socure, so just wanted to bring you in to meet everyone, too.
Yeah, absolutely. No, I've been working for RStudio and with Adam's team for probably around the same time that Adam's been at Socure, so I've kind of seen Adam from his inception of the team, and seen him come this far, and it's really cool to see this. I was a little baby.
I was there before your most recent baby. That's right, that's right, that's right.
And I see you both have your RStudio swag on. I definitely should have been wearing an RStudio shirt here. Limited edition nowadays.
Right. So there are a few other questions on Slido, and one was about Python support. So Quarto and R Markdown products have great Python support. What else, if anything, is needed to have this similar workflow in Python?
Yeah, I mean, I think what's nice about this ecosystem is with Quarto and R Markdown, you can blend the languages if you wanted to, so you can have Python code chunks where you need to, and you can have the R code chunks where you need to. So I don't actually know, so I showed off before in the YAML header of the monitor file where you have to have that R Markdown, or I'm sorry, that RStudio Connect YAML suppression. I don't know if Quarto supports that right now. Somebody can correct me if I'm wrong. I know Quarto is different from R Markdown in a lot of ways, and I don't know if it has that functionality. But otherwise— Waiting for Tom Mock or Isabella Velasquez to magically appear. Yeah, exactly. If we say their name three times, will they appear?
Yeah, but I mean, otherwise, again, you can just use Python to create your alert logic, to pull in data, to make plots. All those things are supported in both Quarto and R Markdown. So you can really do all the same stuff. And now there's Shiny for Python as well. So if you even wanted to build that Shiny app in Python and deploy it, you can totally do that.
It's a great time to be an RStudio user in that regard. I did meet a call earlier, too. I mean, you said Isabelle. So Julia Silgi and Isabelle—is it Velasquez? Zimmerman. I'm sorry, Zimmerman. They gave a talk a couple of weeks ago on Vetiver, which is this nice MLOps sort of ecosystem that you can use in both R and Python, and especially in conjunction with Connect. And there are a lot of great ways to model your monitors with Vetiver as well, both in R and Python.
Absolutely, yeah. And I'll share that link to that meetup recording in the chat in just a second here, too.
Yeah, I'm going to read this next question here for you, Adam. Can this be integrated with Datadog or other similar monitoring tools an org might already have in place?
Yeah, that's a great question. It's one that I should know the answer to, and maybe Gordon can come to my rescue. I'm sure he knows the answer.
So insofar as something like Datadog can—let's come back to this, actually, because what I want to call out next is some of the architecture behind this whole setup, and maybe that will answer some of your questions. But from the Connect side, as long as—if you have data flowing into Datadog, maybe you want to then pull that into Connect. I don't actually know if you can do that. I'm sure there's a way, but I don't actually have enough familiarity with Datadog or the other monitoring tools because I've got my head so deep in this world here. Let's take one more question, and then we'll go to the next section, too.
One was, how would an account manager know what values to use while updating the configuration? Do you provide simulated or historical examples to help guide those?
Yeah, that's a really good point. I think there's a lot of things you could do because R Markdown is so flexible. You could sort of analyze client history and provide suggestions, or even just like—instead of just having a naive default set of thresholds and things when you deploy, the deployment process itself could maybe pull in some data and do some kind of historical analysis and say, these are the thresholds I think kind of make sense. The account manager kind of has to read the reports, make sure they understand the client history, and then it's up to them as well to understand the client's use cases and what they're going to do with all those model scores to then understand, okay, at what point do we need to begin alerting for this client?
So, it really can be—I mean, there's a limited set of things you can do automatically, maybe even in like a model-based way, to set those thresholds. But I think at the end of the day, you want a human being working with the client to kind of make those judgment calls.
Cool. Thank you. Before Tatsu and I disappear backstage, I just wanted to remind everyone that if you do want to ask other questions, if you joined a bit later, we also have a Slido link where you can ask those anonymously, or you can just post them into YouTube or LinkedIn too.
Connect's role in the architecture
Cool. Thank you. Great. Okay. We've been talking about Connect this whole time, but I want to dig a little bit more into how Connect is playing a role here because it's sort of the glue that holds everything together, and we're going to get more in the weeds about what exactly is happening behind the scenes. So, let's talk kind of about the architecture. Again, we'll keep this high level, but just so you know actually what happens when we think about updating a monitor. This is the picture we saw before. So, account managers are looking at monitors. They're looking at this web app. An account manager goes to configure the monitor from this web app, and so now what's actually happening? Well, behind the scenes, the web app is actually interacting, at least in Socure's case, is interacting with a plumber API.
A plumber API, if you're not familiar with APIs, they're just a system for computers or programs to talk to one another, so to communicate, to send data back and forth. So, what actually happens here is Shiny, when you hit that update button in that Shiny app, what it does is it will send a request to this API to say, hey, I need you to update this one particular config file, and the API will say, okay, I'm going to go do that, and the API goes and it updates a set of config files that are living in the Connect storage. So, Connect just has, you know, it's sitting on a server, and the server has storage, and so we have a bunch of config files sitting on Connect, and the API will just go and make those changes to the config file that you requested, or sorry, to the client that you requested it for.
Okay, from there, the config files, the next time the monitors refresh, the monitors then pull in that information from the config files, and that's actually where they're getting their thresholds from. So, we've sort of separated the monitors from that configuration and those settings, all of those thresholds and things, and so that makes it really easy now to make changes to how those monitors are actually alerting.
Config files and JSON
So, what actually is this config file? You know, I said there's files sitting on the server, and that's true. In this case, they're just JSON files. So, whereas previously I said, you know, a configuration could be like a list of thresholds, now in this case, it's essentially just like a language agnostic form of that same thing. So, in this case, it's just JSONs. So, if we're looking at the Willy Wonka company, we might have the wonka.json file sitting here on Connect, and this just has things like the account name, the account ID, the model names they're interested in, what's your reporting window, and then all of your thresholds. So, make that as long as you want, and basically, the deeper you get into data science, the more you realize it's all just JSON and YAML, and that's pretty much what everything converges to.
the deeper you get into data science, the more you realize it's all just JSON and YAML, and that's pretty much what everything converges to.
So, then when a monitor goes and refreshes, what it's actually doing inside that monitor is we have these functions from the automonitor package, and they're basically a suite of API interaction functions that just go out and talk to the plumber API, and then make a request for that config. So, when this monitor refreshes, it's just going to say, hey, I need the latest version of my config file, and the API says, great, here it is, and from there, we can now use that data downstream in our monitors. So, in our case, here we refresh these monitors every night, and so every night, each monitor goes out and grabs its specific set of thresholds and brings those in. So, if an account manager had made a change during that day, that will be reflected the next day in the report.
Why use an API?
Okay, so you might be saying that sounds really convoluted, and it kind of is, although I will say plumber APIs, if you're not familiar with APIs, they sound kind of scary, and kind of magical. Actually, the plumber package makes it so easy to deploy an API that we should all be doing it just for fun. So, but the broader question is, why would we use an API in the first place? And the basic idea is that it really makes it simple to add configs to our Connect storage during deployment. So, you think about this deployment process, again, it's a template plus the config file, and you get a monitor. Okay, so I'll use auto monitor to deploy a monitor, and that sends a monitor to Connect. That's great. But my config file actually gets pushed through our plumber API and then ends up on the server. And then why is that important? Why do we do it that way? Well, it's actually the easiest way to get something right onto your Connect server.
So, now I can just be sitting here. I'm a data scientist. I can be totally abstracted away from where this stuff lives and how it gets updated. I just have this nice API sitting there that I can then use. And then also in our auto monitor package, we include API interaction functions that allow you to create configs, retrieve configs, update configs, delete configs, and so forth. So, all of our data scientists have a nice set of tools at their disposal if they wanted to interact with these configs sitting on Connect. And this also makes it really nice just for programmatic deployments or updates. So, in addition, any other sort of local development or app testing or access by other applications company-wide is made super easy by having this API there.
So, you know, say we wanted to migrate the monitor system somewhere else, and we have that question about Datadog. You know, maybe there are other systems that we want to switch to or even use in addition to these monitors. The question is, well, how would we communicate with those other systems? What are thresholds that we're currently using are? And the answer is just stick an API on Connect. And now all these systems company-wide can talk to that API. So, you know, if we had some other monitoring system instead of Connect, if we just decided to migrate all of our monitoring somewhere else, well, we still have these configuration files that are useful. Account managers still want to configure those things themselves. So, now we can move to a separate monitoring system while still retaining this nice API that then stores our configs.
Swagger docs and API documentation
All right. APIs are also really cool in Connect because they come with Swagger docs for free. So, when you deploy an API, you get a nice set of documentation that comes with that. And let's just take a look at that because if you've never deployed an API, maybe this sounds like gibberish. But this is essentially what the documentation looks like when you deploy an API. So, API has these things called endpoints, which are just like URLs that you can go and send information to and get information from. And so, this is a Swagger documentation that Plumber will build for you, which is so nice when you create an API. And so, now you can, you know, if somebody has a question about, oh, how does this API work? How do I interact with it? Or some engineer somewhere else in the company wants to interact with it, you just point them here and you have this nice set of documentation. You can, you know, in this case, if I wanted to get a client's configuration file, I can use this specific endpoint here and you can open it up and they tell you, you know, what kinds of information the endpoint is expecting. And then you can practice here if you wanted to by adding information to this argument and executing. So, it's super nice. It helps people understand exactly what's happening behind the scenes if they need to.
Current state and future possibilities
All right. I've been talking a lot, so I'm going to start wrapping it up here. But, you know, one, this is, yeah. So, this is kind of our current state, wherever I left you here. This is our current state here of monitoring. There's a future state you could imagine where you don't necessarily have to involve data science in every single deployment. So, I showed before how data science uses the auto monitor package to deploy things programmatically to deploy monitors and that works great. If you even wanted to remove them from the equation a little bit, you know, you could think about providing a deploy feature to your Shiny app so the account managers can go in and deploy new monitors as they get new accounts.
You could even think about maybe having a process that grabs all of our data, looks for new accounts, and if it sees a new account, auto deploys a monitor, sets up a config, etc. So, you can think of a world that, you know, is even more hands-off. We haven't moved there yet, but that's something that's also within the realm of possibility.
External packages on Connect
Okay. One thing I do want to call out with Connect that really helped enable this process for us is the usage of what they call external packages. These are like system-wide packages. So, the idea here is that we as data scientists can build a package, like in our case auto monitor, and then we can create an installation on Connect and that is the version that our API is going to use or that our monitors are going to use and that our Shiny app is going to use. The idea there is that you just have this one nice standardized version that every system is using and you know that it's always up to date because you can go in and configure that.
That's just been a really nice way to make sure that we know exactly what our monitors, how they're going to behave because we are controlling the version of that package on Connect. The documentation does say that external packages decrease reproducibility and isolation of the content on Connect and should only be used as a last resort. I think that's fair. We study YOLO and it's actually been working out well for us in this particular instance.
So, you know, there are really good use cases for these external packages. So, what this means is just that we have now an auto monitor package living on Connect and that is going to be the version that Shiny, our Shiny app uses, the version that our monitors use and then if we ever need to fix a bug or anything, we know that it's fixed everywhere all at once.
Why we keep using Connect
Okay, just to wrap things up, why did we try Connect for monitoring in the first place? You know, like we said before, there's all kinds of great monitoring solutions out there. Even if you or your company is exploring options for monitoring and looking at different vendors, the fact that if you have Connect and you're prototyping today, that has a very high value. So, you don't have to wait, you know, for a system to be stood up and then to go learn that system in order to start deploying and seeing the value of monitors. If you have Connect, have our markdown, you can start doing this stuff today and just kind of figuring out what works.
The whole system, in our case, was sort of security approved, so we could do anything inside that sandbox and, you know, we could just play around, test out, see what works. And, you know, there's just no waiting around for engineering resources or prioritization, you can just start working. Scheduling and user access control is really effortless on Connect. It's super nice to just put something up and then you know who can see it, you know how often it gets refreshed, you can change that really easily. And then also our non-coding partners, like our account managers, can be super engaged and empowered in this whole process.
So, that's why we started using Connect and now this has been a journey that's, you know, longer than my tenure at Socure. And so, the question is, why do we keep using Connect for monitoring even when we have other options at our disposal? It's, well, it's super cost effective, for one. There's no marginal expenditure for hosting additional monitors, for building additional APIs and hosting those, and for pushing out web apps. It's low maintenance, so a well-designed ecosystem can run indefinitely without intervention. I don't think I've actually touched this system or the code in a year because it just runs so well. And when we say well-designed, you know, that's going to include making sure your packages are well-tested. I have not gotten in this talk, but AutoMonitor is thoroughly tested, including all of our API interaction calls. So, make sure you're doing that. But once you do, yeah, it can run really well.
a well-designed ecosystem can run indefinitely without intervention. I don't think I've actually touched this system or the code in a year because it just runs so well.
And then just the extra control. I mean, again, there's no waiting around for someone to go and configure something for you if you need that. You know, if you're a data scientist, you can own this product and can respond to user needs as quickly as you'd like.
Okay. So, with that, thank you very much. Yeah. And Connect has been a lifesaver. And I tweeted out last February just like how Connect has been super helpful just in empowering me and empowering our end users to own their work. So, I hope that you can find a way to do that, too.
I hope that you can find a way to do that, too. And with that, have any more questions?
Thank you so much, Adam. That was awesome. Really, really love the examples you shared here, too. I see somebody had asked on Slido if you would be able to share any of the code that you used or if there was like a public GitHub link that you have.
Yeah. Unfortunately, we don't have anything public in that regard. Yeah. It's all going to be proprietary to Socure and things like that. So, I can't share much. Yeah. I think I saw you sending out links to sort of like the rsconnect package and the connectapi. And yeah. I mean, hopefully, take a look at plumber. And I think with this framework, I hope you can kind of put those pieces together. Absolutely. Yeah. Yeah. And maybe our team can think about putting together like a blog post with some of these examples that you've shared here, too. Would it be possible for us to share the slides with the recording? I think it'd be fine. Yeah.
Okay. Cool. I'll get those for you. And just a reminder to everybody that the recording will be shared at the same exact YouTube live link. So, it will be made available immediately after the meetup ends, too.
But thank you so much, Adam. I see there are a few other questions that Tatsu and I will read here. I'll show one on the screen for you, Tatsu.
How auto monitor deployment works under the hood
Does each monitor instance correspond to an RStudio Connect deployment?
To an RStudio Connect deployment. So, I think there was some extra context added after, too.
Or perhaps better pose, what is an auto monitor doing under the hood when you set up a monitor?
Yeah, for sure. Let's go back. Okay, where'd it go? Way back here.
Yeah. Yeah. So, the main way that a data scientist would use this package is just essentially to call this deploy function. So, what this is doing is it says, all right, here's my account name or account ID, whatever it is. And deploy is going out and saying, okay, I have a template. I'm going to go grab that template. I'm going to create a config file.
And then it takes that markdown template and it combines it with that config file to create a client-specific monitor. And then it deploys that client-specific monitor to RStudio Connect and it also pushes that config file through our API onto the Connect storage.
So, it's, yeah, so it's doing a lot of magic with rsconnect and the connectapi to do things like write the manifest for this file. So, if you've ever deployed to Connect and you've seen that, what it has to do is essentially write out a manifest, which is sort of a list of all of the packages and versions and things that are used in there so that Connect then knows what kind of environment to build for that piece of content. And so, behind the scenes, what's happening is with the help of these two packages, we're writing that manifest and then we're bundling all that content up together and we're pushing that to Connect and that's made possible, again, by these things here. And then we're adding viewer permissions, we're setting the refresh schedule, we're tagging the deployed content, again, all through these nice APIs here. So, it's basically building, it's building that monitor, it's deploying that monitor with all these settings. I don't know if I answered your question, but essentially there's one Connect deployment, like there's one RStudio Connect and then each time we run this deploy function, it's creating a new piece of content and it's pushing that content to Connect for hosting. Yeah, and these tags are the way that you essentially categorize your information. So, Connect can get a little bit crazy with all the reports and things that are in there, but once you tag something, you know that you can go and find it. A tag is just like a label and then you can go and find all the content with that one label. So, if you're looking for all your monitors, you can have like a monitor label.
Yeah, before we move on to the next question, I just wanted to point out that there was some appreciation in the chat about your use of YOLO. YOLO? Really dating yourself bad.
Well, you know, I'm at that age where all of my memes are old, so that's just what I am. And, you know, I kind of embrace it and it's partly tongue-in-cheek, but also I'm just an old guy, so. Really embracing bad life. There you go.
I went to grad school in Northern California at UC Davis and it's in YOLO County and it was just the greatest thing. I tried to just share a link on Connect tags. It looks like a very long, weird link there, so maybe we can help find a better one right from the top. Okay, yeah, you know, the Connect API package, I believe, is still under active development. I did share that too. I shared that into the chat too. Okay, cool.
I see there was a question a bit earlier from someone who doesn't have access to Connect, but asked if they could use SageMaker with the Shiny model monitoring. And I just wanted to say that next Tuesday we're actually going to have a few of the product managers from our team here to give a few lightning talks. And one of the people that will be joining us is James Blair, who's our product manager for the cloud integration. And so I think that would be a really great question to ask him as well.
So I wanted to share just while I'm thinking about it, if you are curious where to find out about future events, you can look at this calendar link here. And I'm bad at multitasking right now, but I'll share it in the chat in just a second too.
Updating R Markdown templates at scale
But another question over on Slido, Tatsu, I'll put on the screen for you to read.
What do you do when your R Markdown template needs updating? Is that handled via the app too?
Yeah, great question. So yeah, what happens if you want to add a new kind of plot or something to your template? Yeah. So what we did in this auto monitor package, it's not just for deploying new content, but you can also redeploy or update existing content. So if you wanted to make a change to your template, again, that template's just like a single file that lives in your package. So you could change that template, add new sections, whatever it is.
And then what's the best way to do this? In this framework, I think what I would probably end up doing is because we've built in, yeah, we built in all these nice API interaction functions. If I had to update all those monitors, what I think we could do here is go out and grab all of the existing configs. And then that contains things like account name, account ID. And now I can use my deploy function, which was specifically written to either deploy new content or update existing content. And I can go say, okay, from this list of all of my existing configs, all of my existing clients, I want to grab the account name. And then I want to redeploy with this new template. And that would just be done in a loop programmatically. Might take a little while, but it would actually be doable.
Parameterized R Markdown as an alternative
I'm not sure I see any other questions yet in the chat. Do you see any others, Tatsu?
I see Thomas asked a question. Did you look into parameterize our markdown reports where you could schedule and connect with new or additional sets of parameters?
Yeah, definitely. And that's something that is sort of adjacent to this idea here where maybe you have a template and instead of writing out separate R markdown files for each client, maybe you just have a set of parameters in the YAML header. So if you're not familiar, if you parameterize your R markdown, that just means your YAML header is given a set of configurable parameters, which then you can pass downstream to all of your R code.
So that's definitely an option. I think there are some limitations with how we're doing it here. So I think one limitation is a parameterized report is like one single monolithic piece of content on Connect. But I don't believe that you can very easily control the access and the scheduling for each individual variant that you render. So what you can do on Connect is once you deploy that parameterized piece of content, you then can open this nice sidebar, and it gives you a way to change all of the input parameters to those reports. So you want to change the account ID, you want to change the reporting window, you want to change the thresholds, and then you can save what they call a variant. So it's like a new version of that report with those new parameters. And those all kind of live together in the same piece of content with a drop-down menu where you can go and select which variant you want to view. And I think there's a lot of merit to that. I think in this case, we have too many clients, we have too many monitors, and it's a little bit time-consuming for all of our account managers to go in and have to go in and search for the clients that they want to find in that big list of variants. So it's definitely a good option if you have a handful of different versions of the same report you want to run. I think in this case, it didn't make sense for us.
Impact on data scientists and account managers
Can I selfishly ask a question, right? Sure, I would love for you to ask a question.
Yeah, absolutely, selfishly. Well, I talk to a lot of folks in consulting, right? So it's really great to have this example out there now out of, again, like fantastic talk. And my question is around when you came up with this, right? You have these parameters for model monitoring that your account managers can leverage, right? What was it kind of like before you developed that piece and what has that change looked like? You said it empowers the account managers. Has it created better relations with your clients? What does that look like for you?
Yeah. So your first question was how we kind of hit on this, right? Yeah. Shout out to Gordon. You're still out there. I think this was sort of his brainchild. And I'm sure probably Jonathan too, but I sort of stepped into his shoes after he left Socure. Yeah, I think like this journey here that I kind of walked us through was our journey. And so like this was the pain right here that everyone was feeling. And it was sort of the thing where like, well, people just stopped paying attention to their alerts because it just became too much. And when you have too many false positives, you just assume everything's a false positive. And then you miss the really important stuff. And so we built this whole system summer of 2021, more or less, to start thinking about how do we actually allow this self-service?
when you have too many false positives, you just assume everything's a false positive. And then you miss the really important stuff.
And it did kind of come as a result of just all these tickets. So this is like, this is like true story here. It was just so many tickets coming in and our poor data scientists were just overwhelmed and it was just too much. So yeah, so then we deployed this. And honestly, like I said before, we haven't really even had to do a lot of maintenance on it.
What this did was it basically reduced our tickets from a million to like, it's not just alerting noise, it's ticket noise. And you get overwhelmed and you're just like, okay, I spent all day updating these files. I could be spending my time somewhere else doing something more valuable for the company. And so this cut down for us on this long feedback cycle where it's like, hey, update this thing and let's just like, let's hope it works well for me. And so it wastes data scientists' time and it wastes account managers' time because in the time it takes them to write a ticket, they could have configured the thing anyway. And then they, if they have to write a ticket, they have to wait for it to happen. And now they don't, they just go in and they just update it and it's done.
Yeah. And like, this is the real deal like this. So this is like, every now and then we'll get new account managers coming in and they'll ask a question and they'll submit a JIRA ticket and I'll point them to this web app and they'll say like, this is super handy. And like, yeah, it's just, it's super easy for them to use. And yeah, it just, it kind of removes us from that whole, it breaks that whole like nasty feedback loop of involving data science where we just,
frankly, we could be spending our time doing other things. Yeah. Not to mention like human error and the shortness of our attention span, right? It would totally just explode. So yeah, I love it.
Yeah. And again, like it, like a feel like this, if you just look at this, this feels convoluted. But like when you step back and you realize, oh yeah, this API like has a really important, like it's this sort of like central, this central element, this whole thing revolves around this API because now it, you, you put a thing up in front of it, you put an app in front of it and now non-coders can use the API or you have API interaction functions and now data scientists can use it or you need to expose it to engineering for some reason and now engineering can use it. And it's just like, it's so cool that like that one thing can control all these processes.
And it's just like, it's so cool that like that one thing can control all these processes.
Yeah. I love seeing how all the pieces fit together. And it was reminding me of the, the bike prediction example we share in our solution site. I just put into the chat here, but when you like see everything individually, like tied and then tied together, like scheduling our markdown document, feeding that into an API and then a Shiny app, you get to see like the big picture there. So I'd recommend checking out that example too. Just scrolling through to see any other questions I might've missed. Let's see. I don't think so. Is there a question, Adam, that you were expecting we were all going to ask you that we didn't?
Yeah, I guess. Yeah, I know I called out like the APIs can feel a little scary. So I was expecting kind of more questions around that. But I do think that it's like plumber is super well documented. If you are afraid to ask plumber is super well documented on the website. And I'm sure you shared that link already right there. Yeah. I think that's a great question. And I'm sure you shared that link already, Rachel. So yeah, go check it out if you're sort of curious about how the API works. And if you just have, you don't have to deploy the API to connect. You can sort of run it locally on your own little machine if you wanted to just try it out. And it's just super fun to play with once you get the hang of them. And it's so easy. Like you write everything in our code if you want to. And so it's like you write our functions, and then you just add a couple little like comments with decorators in them. And then plumber takes care of the rest.
Understanding APIs
It's just a great, great topic to cover because I did just see an awesome tweet from somebody from the community who said, What is the best way to describe APIs to a five year old and one that does not have computer scientists as parents? And so just curious, like what was one of the best resources for you when you were just getting started with understanding APIs?
With understanding APIs? Gosh, I forget why I started reading about them in the first place. But just because it's like, it's this word people throw around all the time. Sometimes they use it to mean the code you write to interact with like a lower level library, that that's sometimes called an API. Sometimes they mean, you know, something that a request you make to a computer server in order to like, get some information back. I think what was just helpful for me was thinking about it as, yeah, just like, here's a standard agreed upon way for two computers to talk to one another. So computer A wants some information from computer B, and they have to speak the same language to understand what it is I'm asking and what it is I'm getting back.
And the API is just sort of like, it's just that layer in between that allows the computers to sort of speak the common language. That was sort of the light bulb moment for me. It's like, oh, yeah, okay. It's just a structured way for computers to share information. If you go to a web page, you're essentially making an API call to a server. Like, hey, I want to, like, give me the contents of this page. And the server says, okay, great, here's all the HTML. And then your browser says, here's the page. So that's kind of like, yeah, thinking about it in terms of just like going to a website also helped me a lot.
It's just a structured way for computers to share information. If you go to a web page, you're essentially making an API call to a server. Like, hey, I want to, like, give me the contents of this page. And the server says, okay, great, here's all the HTML. And then your browser says, here's the page.
Thank you. That was really helpful. And I know we are definitely way over here on time. But I think we got to answer all the questions. So thank you so much, Adam. YOLO. I did just want to also mention, if you were watching this and thinking, like, I don't actually know what Connect is yet, or I'd be curious to talk to someone from our studio, if you'd like to talk with someone like Tatsu, who's on here today, I've included this link here. I'm always so amazed by how many teams or people from teams don't yet know that they have these tools available within their organization. So always happy to help connect you with others across your company, too. I mean, especially companies with thousands of employees, it's really hard to know who has everything. So if you ever are curious, feel free to reach out directly on LinkedIn to me, or you can schedule a call to chat with our team there. But really appreciate you all joining and all the great questions. Thank you so much, Adam, for sharing your experience, too. Awesome. Thanks for having me. I enjoyed it. Thanks, Tatsu.