image: thumbnail.jpg
Posit Meetup | Shatrunjai Singh, Aetna | R In Insurance
Presentation by Shatrunjai Singh
Abstract:
An algorithm is only as valuable as its adoption. Speed to value, repeatability, and low-cost solutions can dramatically reduce software and services budgets and free up valuable dollars for other activities. Open-source tools such as Shiny (R) and Flask (Python) have enabled the creation and deployment of data science-based web applications convenient and manageable. In the healthcare data science world, we routinely wrap sophisticated statistical code into such web-based point-and-click solutions. In this talk, you will learn about real-life examples of how one can rapidly operationalize intricate algorithms using web app frameworks.
Bio:
Shatrunjai ‘Jai’ Singh is a Lead Data Scientist at Aetna, a CVS Health Company and specializes in data mining, predictive modeling, and data visualization. His work has received several awards including the American Heart Association and the Epilepsy Foundation. He won the Tableau Chart-Champion in 2016 and was included in the ’40 under 40’ for innovation by LIMRA International.
Q&A here: https://community.rstudio.com/t/meetup-recording-operationalizing-algorithms-using-shiny-and-flask/102463
Transcript#
This transcript was generated automatically and may contain errors.
very much, Jay, for your time today. I really appreciate it. And I'll turn it over to you.
Thank you, Rachel. Let me quickly share my screen.
All right, perfect. So my name is Jai Singh. I work in CVS, Aetna. Aetna is a big health insurance company here in the US. And we were acquired by CVS, which is a big pharmacy company here, also in the US. I work in data science. I've been in insurance for the last six years. And I've been in data science for more than a decade, based out of Boston. And today I will be talking to you guys about how we have been using Shiny to build up these rapid prototyping frameworks, which help us take data science and operationalize it. And I'll be showing you some of the apps that we have been using in our day to day working.
But before I begin, as always, a quick advertisement, if you are looking for a new job, or if you are interested in opportunities in data science, please do consider us. We are a fortune five company, one of the five biggest companies here in the US, great benefits, work life balance. The other thing is that executives, they have a very strong data hole, they love making decisions based on data. That's why there's a lot of visibility on data science. And we are hiring very, very rapidly all across the US. If you are interested in opportunities anywhere in the US to contact us, we are hiring from all different roles, from analysts, all the way up to senior directors. And if you're interested, you can always Google CVS data science and you look at the opportunities, but a better way is just email me. And I can shortcut you through through most of the process and get you in get your foot in the door pretty quickly with interviews. So do consider us if you are looking to make a switch or if you're interested in data science at CVS at now.
Rapid prototyping and web app frameworks
Alright, so rapid prototyping, essentially is taking what whatever analysis you're doing. And if you feel like you will do it more than twice, then you build a web app out of it. And this tends to help you in a lot of different ways. One of the biggest things it helps you with is that now you can share the minimum viable product with a bunch of different people and they can iterate through it. And that will help you get the most efficient and optimized analysis. And it makes it very reputable, you can make a GitHub page. But if you make a shiny app, it's easier for people who are not R or Python, or just in general, SAS users to play with your analysis and quickly get to the endpoint faster. So it standardizes a lot of it.
So that's why rapid prototyping, you would see that it's it's a very common theme in different industry paradigms. People in statistics will be familiar with Chris TM, which is cross industry standard process for data mining, where you start with the data, you perform data pre processing, you build a model, you evaluate how well the model is working, you deploy it. And then usually you go back to the business and they tell you everything you've done is completely wrong. And you start from scratch. So Chris TM, design thinking, agile, you'll see they're all very circular. And essentially, what they're trying to do is get to the minimum viable product faster, and then test and learn.
And to do this, you can use all these different frameworks. In data science, you can build web applications. And if you look at the history of web applications, it started with the HTML in the 1990s, probably some lonely grad student in a in some computer science department made this up, probably not. And then over the years, you saw that it has evolved. And now we got in close to 2000, we got these web applications made in Java. In 2014, we got our shiny, which changed a lot of how web apps are made. It was developed by a Cho Chang. And essentially, it removes you from trying to have expertise in HTML, CSS and JS. And you can actually build your applications, which are data science heavy. And you can make them with an interface, which is very, very pretty, a GUI, which looks good and easy to navigate. Now, over the few years, now you have a bunch of different web app tool frameworks like flask, dash, streamlit. But I still prefer shiny. And I'll tell you why.
Now there are trade offs. So you can consider building your web applications in different frameworks. So you can use flask, which is Python based. And you can sort of mix flask with JavaScript. To make a web application. This is throughout a lot of different companies, especially if you see things on websites, this is they use it with Django. And this is this is the way they use most of the web applications. Dash is pretty popular. Tableau is more for dashboarding. And I'm sure you guys are aware of how Tableau dashboards have been made and employed throughout the world. And then obviously, they're shiny.
Then you can compare all of these different frameworks on all of these different measures. The three ones that I find very important are just stack overflow support. So if you are a data scientist, you know how much how important it is for you to have support and be able to find help online. And I find that shiny tends to have a lot of help online just because it was the first to market. A lot of people have failed doing the same things that you are trying to do. So that tends to help. It is popular with data scientists just because R is one of the two biggest languages for data science. You will speak the same language and somebody who comes after you will be able to maintain the apps that you have created. And finally, it's free of cost, which is pretty important. You don't want to pay $3,000 for a software package, which in the end, if it does, if it doesn't add to the free tools that are out there, then you've essentially cost the company 3000 bucks. So that's one of the areas that I think shiny tends to shine or do very well overall.
You don't want to pay $3,000 for a software package, which in the end, if it does, if it doesn't add to the free tools that are out there, then you've essentially cost the company 3000 bucks.
Now, if you're not familiar with, when do you use web apps, which I highly doubt if you're in this meetup, then you would use web apps when you want to revisit the same code for different projects. So if you're trying to do something similar over the different projects, you can make a web app out of it. If you want to introduce a new technique, I've seen this work pretty well. One of the apps that I'll show you today is the comorbidity analysis, which is a technique that I had published a paper in academia, but wasn't used in industry too often. And making a web app tends to help with that. It leads to faster adoption of whatever you're trying to sell. And finally, the most critical one is if you're trying, if there is an analysis that you do that has multiple steps, and that analysis is pretty common. So multiple groups within your organization are using a similar framework, and then getting different answers or different variations of the same answer, and then presenting it to the business, this leads to a lot of confusion. So if you standardize your whole data science process, and you build a web app that everybody can use, then that tends to shorten the time to analysis. And it also helps in standardization overall. And usually, web apps are more useful when you need slightly more than just dashboarding. If you just need dashboarding, then the best things are Power BI and Tableau.
Build vs. buy decisions
So what do you do when you're trying to make a decision on whether you should build a web application in house? Or should you outsource it to somebody else? Now, there are seven. Now there are seven different things you can consider. Cost. So if you're trying to buy it, that is you're trying to get somebody else to build it for you from outside the org, it will be more costly. The customizations, you might be able to get some customizations, but more additional customizations will cost you even more. And then the biggest one is knowledge gain, you will not have the knowledge gain somebody else will. It'll save you on man hours, build time and support. If you build it in house, which Shiny allows you to do build these rapid prototyping frameworks in house very quickly, then it will be cheap, essentially be free will only be the man hours that the data scientist requires. It'll be very customized. And you will have knowledge gain, it will take some time for your department to build these apps. And you are the support if you build the app. Obviously, it requires majority of your time. So these are some of the downsides of whether you should be building an app, or you should be buying a solution from from outside.
Apps built for CVS
All right. Having said that today, I will show you some of the apps that I've built for CVS. And how have they how have they been used? Some of the different ones are the first one that I'll show you is called the clustering and profiling app. It's a web application that does market segmentation, which is pretty standard, you're trying to find groups of people that are similar to each other in in a diverse population. I'll show you a comorbidity analysis app, which is useful for finding comorbidities. Now, in health insurance or in health care, comorbidities are when you have one more than one health condition. So suppose you have diabetes and hypertension, then you have two conditions. So it's a comorbidity. I'll show you a propensity score matching app. So this comes from causal inference. So if you do not have a randomized control trial, so it's not randomized. So you have to do quasi experimental methods, like matching.
Essentially, what you're trying to do is create a synthetic control group. And to find these controls who are similar to your test group, you have to match them on different criteria, the different features. And to do that, you can use this app called propensity score matching app, this is more causal inference. I'll also quickly show you the equal tool that I built for a competition within my company, it's called the competition was called ABC. And they judged you on how innovative your app was, how much money will it save the company and how good looking you are. So obviously, I was in the top five. But what equal does is equal stands for eliminate quantitative unfairness in algorithms. Now you might have heard of how predictive models might be propagating bias in algorithms. So because algorithms or models are built on real life data. And if there is bias in in the data set, there are historical biases in the data set, a model will not only propagate them, it will accentuate them. So because CVS is very big on fairness and equality, we have checked all our models to make sure that we are equal, we are fair, and we are unjust. And this tool is being used to do some of those things. We also have a topic modeling app that I will not show you today, which uses NLP. And it does some topic modeling on notes that people get from from customer care.
Clustering and profiling app
So the first app, which is the clustering and profiling app, the big problem was that multiple teams were performing customer segmentation. And they were all using slightly different methodologies, but slightly different variations. And they were getting similar results, but not quite. And then they were presenting it to different business units, which was causing confusion, confusion. And this comes from the fact that people have been trained in market segmentation, but everybody uses a slightly different route. They present their results slightly differently. People are either R users, Python users, or SAS users, and they use different variations, they use different little things like imputations, or capping, they're they're slightly different. And the end result tends to be different. This confuses everybody. And in the end, a department like marketing who's trying to figure out what segments exist in, say, a population in Boston, if they get different answers from different business units, they have to recalibrate very often. And this costs a lot of time and money, obviously. So this was the big problem that we were facing.
Now, the clustering and profiling app helps you get to clustering insights faster. It's a web application developed in, in R Shiny. At CVS Health, we have an RStudio environment, where you can publish these apps. We also have support for Python and SAS as well. And the way this app works is it uses four steps. It performs exploratory data analysis, it does the actual clustering, it also gives you insight generation. So it'll look at your results, and it'll tell you what are the groups, how do they look like? It's again, again, it's agile framework, you can rapidly iterate through multiple things, you can change whatever you like. And this is mainly for people who are familiar with clustering and how this market segmentation analysis work. It's not for people who don't have a background in data science, because then these results can they can get a little confusing for you.
And just to just to tell you how it has changed the way we do market segmentation. Before this app, we had to do a bunch of these different steps. And now we can rapidly iterate on all of these different steps. For example, for EDA, we had to do all these missing data renaming and correlations. And now it's a once click EDA. Similarly, we do similarly for clustering, profiling, insight generation, summarization, and even subsegment analysis. It's now all semi automated, and it's way quicker now.
Alright, so this is how the app looks like. You can come in, you can take a quick tour, which will introduce you to the different parts of the app. Essentially, what you do is you can import your data, you can explore it. So you can explore
your data, get summaries, missing values, correlations, you can find clusters in your data, you can use different methods for clustering, we use, we prefer create prototypes for mixed data. You can then profile the clusters you found. So for example, you found these three different clusters in your data set, then you can profile them, you can, you can quickly find that these clusters differ in age and their risk. And then you can also get more detailed insights, you can find out what are the key variables that triage this risk. So the data consists of people who are either below 50 or above 50. If they are below 50, they go in the first cluster, if they're not, they can be further bifurcated by risk. And then they can go into cluster two and three. So you can use decision trees to form these clusters.
To enable, so the one thing that I've learned by making these apps and launching it to a small audience is, the more explanations you can give, the more intuitive it is, the better. And that's why using JavaScript in here for all of these different helper little boxes, that tends to be very useful. And just giving them directions up top, it all, it's very useful for people who are not very familiar with your app. Also uploading a toy data set is highly recommended. If you've not, if you just directly put in an upload your own data, then people get confused. So I put in like three or four different toy data sets that you can use from different sources. I'll show you clinical data since I work for Aetna. So let's look at this data set, which is just it's fake. It is a data set of members with different diseases. And different characteristics. And you can pick the different variables that you want to include in your analysis. Up here, you can quickly explore the data and get summaries. You can find what percentage of zeros exist within your data set, you can find what how many different what is the different types of factors within each variable, and what kind of variable is it. And it'll give you like missing data summaries as well.
Now, this is pretty powerful, you can quickly get summaries, you can download all of this in either of the formats that you want. So you if you want a PDF, you can quickly download this as a PDF. And it'll give you all of these summaries in whatever format. It'll show you correlations very quickly, you can find what are the variables that are very correlated with each other. So if you spend days in an inpatient facility, you will obviously have very high medical costs. So that's, that's why you have high correlations. And you can view your raw data, whatever you have uploaded. Also, you might want to remove variables which are very highly correlated. So then you can remove those variables, right here going forward, they'll be excluded from the rest of the analysis. So this is how you explore data.
Then you can go into the clustering analysis where you say decide the number of clusters, a very common data science clustering 101 technique using the elbow method. And then you can choose whatever, whatever method you want for your clustering algorithm. And just quickly click to cluster. And for example, if you use three clusters, then it gives you this result. And you can also change clusters according to business needs. So if the business says three is too little, we need something between four to six. And you and if you've done this before, you know that market segmentation is not a pure data science exercise. It's a mix, it's a mix between what the business wants and what you can give them. So you can very quickly iterate on what you want.
Also, we've seen that the variables that we have, some of them are continuous, and some of them are categorical. Sometimes you might want to weigh the continuous variables a little more. Things like age tend to have more information in them, say age between 18 to 90. Then binary variables like do you have a Ferrari or not. So we do want to exploit that higher information content in continuous variables. So you can also play around with the weightage that you put on the different variable types. So for example, we chose three clusters. And these are the three clusters that you've got right now. You can download all the results at this stage. Or you can go on to the next part, where you get some more insights into how your clusters look. So this gives you the distribution of clusters will give you either the count if it's a binary variable, say gender male, it's a binary variable, it will give you the count and the percentage. So 59.3% of the members in the first cluster are male. In the second cluster, you have 55% of the members are male, so more males. And in the last one, it's more females, only 15% of them are males. And they'll give you the average risk or what percentage of them have asthma. So 6% of cluster one have asthma. So this is giving you insights very quickly.
What is more interesting in is how they compare to each other, which is what we get in the next tab. Now it'll tell you for males, the first two clusters are very heavily male, the third one, not so much. Similarly, for age, the first two clusters are more older, the third one is younger. So now you can see very quickly you're forming a profile for these clusters, you're, you're saying that the first two clusters are male, older, the third cluster is more female and younger. Finally, you can also get the profiles of these clusters generated. So from the z-scores that I've, I was showing here, you can very quickly find the top variables that define a cluster. So cluster one tends to be people who are obese, they are either they're more rural, and they have metabolic syndrome and so forth on. And if you go back and change your clustering algorithm, all of this will change very quickly. So suppose you wanted four clusters, you think the three clusters you have are not very good, then you can just come back and very quickly change all of this results, and you'll have four clusters. So you see iterating through your data set now becomes very easy. It's very intuitive.
So now you have your four clusters, you've done everything. And now you want to understand your clusters. This is where the decision tree, the CART algorithm comes in. And what we have done here is we've we're trying to build a decision tree trying to predict what are the different clusters and how do you get to them. So the different clusters are shown in different colors. So you have cluster number one, two, cluster number three, and cluster number four here. And if you have an inpatient score of greater than 35, age less than 59, then you will fall in cluster number four. So now you've got the rules as well. So now you can identify, there are four clusters in my data. And these are the rules that allow me to get to a population which will be cluster number four. It'll give you variable importance based on the CART model, it'll also give you the rules written out. So it'll tell you what percentage 36% of the members will follow this rule, and so forth on. So this tends to be very powerful.
Finally, you're done with your analysis, what it does is now it download an Excel file with all the results and a formatted PowerPoint. So not only you've completed your analysis, it'll give you a formatted Aetna PowerPoint with Aetna, custom fonts, charts, everything. So now you can just take this analysis and show it to your business unit, and they would know what you are talking about. And this tends to be very powerful, you've completed a segmentation exercise in minutes, which usually takes our business units, which were working alone with either their own code and iterating through different steps, weeks to do.
And this tends to be very powerful, you've completed a segmentation exercise in minutes, which usually takes our business units, which were working alone with either their own code and iterating through different steps, weeks to do.
Just to show you how the final PowerPoint looks. So this is the final PowerPoint that it has downloaded, it shows you all the different steps, it shows you how many clusters were found, and so forth on the methodology that you used. Finally, we did put in a feedback form, we wanted to learn from the audience and get better. So we are trying to measure things like customer satisfaction score and net promoter scores, and also trying to get more feedback on what what changes would they want.
So let me get back to the PowerPoint.
So we've used this for different projects within the org. Some of the examples are here. The one project that I will talk about is this meditation app that we had done a project with. So this meditation app was sent out to a bunch of different people. And apart, some of them got engaged, that is they did download the app and they signed up. So we wanted to see what are the different segments that exist within the people who've downloaded this meditation app.
And we found using the app, we found that there are four different segments. And using the insights part of the app, we found that three of those segments, so you can use four variables to understand how these segments are made. The first is gender, the first three of the segments are female, they are 8800 100% female, the last one is male, then it's risk. So in health insurance, you define risk as how sick a member is how risky he is in the next one year. So within the females, now you can bifurcate on risk. So one of them is high risk to our low risk within the female low risk segment. Now you can bifurcate on income, one of them is higher income, and one of them is lower income. So now you understand that the people who are downloading this app are falling in these four buckets. And in the next iteration of the app, you can send out emails which are more customized, you can target females who are older and lower income, compared to males who are low risk and low income, and you can send out these creatives accordingly. And hopefully you'll get a more higher engagement rate. So this is how this app quickly led to us getting a more engaged population, more people meditating in general, and you do see impact.
Measuring app success
Now the app was a hit. And the way we can call it a hit is we can measure it on three things, we can measure it on what users tell us. So we can use things like how many people like this on our internal social media page, what was the customer satisfaction score, so forth on us. We can also find out things that the customer do not tell us, but we know. So we can we put little trackers within the app, which can help us count the number of people who view the app, the number of people who uploaded their data set, the number of people who ran their analysis, and the number of people who downloaded the end result. And we can use that as a metric to say, hey, this app is actually making a lot of impact because 81 people have downloaded their entire project results, and I've used them. And then there are the unknown unknowns, which is people might have used this app and got inspiration to do other projects and quantifying that impact is a little harder.
Comorbidity and multimorbidity analysis
So the next app I want to show you is a comorbidity or a multimorbidity analysis app. Now, as I said, a comorbidity is two diseases. So if you have two diseases, it's comorbidities that you have. If you have more than two diseases, it's multimorbidities. And the figure on the left, it shows you that if you have a heart condition, or if you have cancer, then your life expectancy tends to decrease by the number of additional health conditions you have. So if you only had a heart disease, your life expectancy after you first got heart disease was say 20 years. Every additional disease, if you got diabetes along with it, then it'll start decreasing your life expectancy. And on average, your life expectancy decreases by two years for every additional condition you have. So if you only had heart condition, and now you've also got diabetes, then your life expectancy just decreased by on average 1.8 years for every additional condition you have. So multimorbidities are costly. And comorbidities are costly. In fact, we also looked at our own internal data and we found the same trend. The more diseases you have, the more number of additional diseases you have, the more costly you get in the next one year for us. And obviously, for us, our main goal is to make our members healthy and save money in the process. So trying to find this population, trying to figure out what are the most common multimorbidities and what are the most expensive multimorbidities, what are the most impactable multimorbidities is very, very central to our whole business.
Now, the analysis I'll show you uses something called market basket analysis, or association rules. People in retail are must be familiar with this analysis. Essentially, what it is, is if you had these, if you've, if you're sitting at Walmart, at the cashier's register, and you see people make these different transactions. So in the first transaction, they bought an apple, a beer, cocaine or sugar, I'm not sure what and then meat. And these are the different transactions that they had, then you can calculate three essential metrics. The first is support. So four out of the eight transaction contained an apple, so the support is 0.5. If then you can also define confidence, that is the people who bought an apple and a beer among the total number of people who bought an apple. So here, you would just try to find how many people who bought an apple also bought a beer. So that is a confidence. And finally lift is just how much more likely are you to buy a beer if you bought an apple. And if you have a lift greater than one, that that means there is a positive association. So you're more likely to buy an apple if you bought beer. And if a lift of left less than one is negative association, that is you're less likely to buy it. Now, you might have heard of stories where they've used association rules like diapers are now kept with beer. And it's because
association rules show that people on I think Tuesday or Wednesday night, they tend to go out to Walmart, and they look they will buy beer with diapers more likely these are parents who are just trying to get a diaper and they'll be like, okay, I need a beer too. And so these are used either they put the things together, or they might put them on opposite ends of the store. So if you go to Walmart, if they found out that you need Apple, if you need beer and diapers, they'll put beer at the front of the store and diapers at the very back. So then you will walk through the whole store and buy more things. So that's how they use association rules in retail.
Now you can employ this the same logic in healthcare as well. You can define support as the number of people who have disease one and disease two overall. So if you have 100 patients, how many of them have hypertension and diabetes? Confidence is how many people who have the two diseases divided by the number of people who have the disease of your interest. So of the people who are diabetic, how many of them also have hypertension? And the lift is just then how much more likely are you to have hypertension if you already had diabetes, if your lift is more than one, then you're more likely if it is less than one, you're less likely. And so this is a paper that that uses association rules for comorbidity analysis. And there have been a lot of different papers that have used it including myself, I published some research on using association rule in comorbidity analysis.
So, again, in the app, what we did is we put in our generic exploratory data analysis part where you look at missing data, you do correlations, things like that. But specifically the use cases that you can use are you can answer questions like what are the most common multi morbidities in the population? What are the most expensive multi multi morbidities in the population and so forth on. So let me quickly show you the app.
Just takes a second to load. In the meantime, if you guys have any questions, please feel free to put them in the chat. And I hope to have some time in the end for questions as well.
Alright, so here's the app. Again, it's a generic again, it's you can you can use this app for all of these different parts of multi of comorbidity analysis. You can also find literature here on PubMed. And if you have any questions, you can send them to me. I did upload a toy data set again, and you can select the this toy data set is just all the it has like 1000s of members, along with all the different diseases that they have. So if you have say HIV, AIDS, then you'll your row will say one, otherwise it will be zero. So it's just binary data for all the different members.
You can do exploratory data analysis. Find all of these different things in here. Very similar to ones that I've shown you before. And then you can very quickly find things like what are the most common comorbidities in my data. So you just sort by support. And you would see hypertension with hyperlipidemia. So if you have more cholesterol and your blood pressure is high, that tends to be the most common. In fact, 58% of the members we see this combination. So this is a combination of two what if you're interested in a combination of three or more diseases, what are the most common three or more conditions. And again, you can very quickly sort on this and you see that diabetes with hyperlipidemia and hypertension is the most common three diseases and so forth.
And you can keep very quickly change what what you want to find overall. So it gives you what are the most common multimorbidities overall. Another question you might want to use it for is in our most expensive members and by most expensive we define them as the top quartile of all the medical costs in the next one year. So if you are in the top 25% of cost in the next one year, what conditions would you have? And suppose we want to see conditions from two all the way up to five, then very quickly you can find if you have ischemic heart disease with hyperlipidemia and hypertension, you will be very costly for us in the in the coming one year.
Sorry, I apologize. So you'll use confidence. And the most costly conditions would be heart failure, renal failure, in general. So renal failure tends to be you have to go on dialysis. So it's more expensive. And if you had had heart failure, more more likely you're more likely to have certain heart procedures which are very expensive. And that's why it's coming on top.
So so this is an example of how you can quickly use this. Again, you can download all of this as an Excel file. Or you can just print it as well. And this tends to be very powerful in quickly finding out what are the most important comorbidities that you have.
I only have 20 minutes left. So I'll skip this the other parts. I do want to show you one cool thing. So I do have some documentation. But the cool thing that I added in here is a chat bot. So often people have simple questions that you have to answer again and again, a way around it is to use a chat bot function. I use a Dialogflow API. So Dialogflow is now acquired by Google. And they use natural language processing. And you can specify certain web pages that they can scrape for information. And then they look at keywords and give you the replies from those web pages. You can also input your own answers that you have. So you can even chat with it actually.
so you can ask simple questions that you had, for example, what's the maximum size of files that I can upload? What what do I do if I get this error, things like that, you can make a chat bot, it's easier than having a document that people have to scroll, scroll through. And then as I said, you can add admin tracking in here, I can track what people have been using it and how long have they used it for how many hours they have done.
So, so, so this is one example where you can look at comorbidities related to health conditions, you can also look at drug combinations. So if you had a data set with different drug drugs, so drugs for hypertension or drugs for anticoagulants or something, then you can quickly find find out what are the most common drug combinations that people have been using? What are the most expensive drugs that people have been using? And that's pretty powerful. Right, yeah, so you can use what, what are the different drug combinations that they are using? What are the combination of diseases that most expensive cohort of members have and so forth on?
Propensity score matching app
Moving on to the propensity score matching app.
There's quite a few questions coming in and the Slido as well. If maybe we could cover those and some of those in a few minutes, too. That would be awesome.
Sure. Yep. I think I'll be done in 10 minutes. All right, so propensity score matching app, as I said, propensity score matching is a tool that you can use to help you find out what people have been using and how long have they been using it. Alright, so propensity score matching app, as I said, propensity score matching is a is a causal inference technique. So if you do not have a randomized control trial, and you only have a treatment group and you want to estimate the average treatment effect, then you would have to find people who are who can be a synthetic control. And then you have to create for every member in the treatment group, you have to find a matching member who's very similar. And you can use him as a control.
So say if a member was living in a metropolitan city, he was a middle aged male, his candy based candy crunch score was 50 in candy crunch, then you would have to find a control member who had all of those similar similar characteristics. So to do that, you use something called propensity score matching. It's a very popular technique in quasi experimental trials.
And you can use this analysis to do different things, you can find baseline characteristics. I apologize if this sounds a little foreign to you. But this is more for experimental analysis for it's very useful in when you're trying to run quasi experiments. So the usual process of running these experiments is first you have to find baseline characteristics. That is, at the baseline, how much are my treatment group differ different from the control group, then you match them, you find people who are similar in the treatment, and you find matching controls. And after you found them, you try to quantify have I been able to remove all the differences that were there in before matching. And then you also sometimes perform something called the sensitivity analysis. So this is something that we, we tend to run a lot of these quasi experiments, and we have to do propensity score matching quite often. And for that we use this analysis.
Yep, and this is an app that created for propensity score matching, you can perform exploratory analysis, you can find covariate distribution, perform the propensity score matching, you can iterate the analysis and download formatted results, again, upload your own data or use toy data set, and find all of these, you can find the missing data, and so forth on. And then you can finally do the propensity score matching, which which is an allows the whole analysis and download your final results. Again, I also have a propensity score matching chatbot, very similar to the one that I showed you before this.
And in the last five minutes, I do want to talk about one other tool, which is equal, which
sorry to interrupt, but just because you were showing that chatbot right now, I know there were a few questions on that. And one of the questions was, can we have real time customer service through this tool? Or is it just based on NLP?
And then right now the chat? Yep, sorry. Sorry, go ahead.
And then also just if you could maybe share a GitHub or resource for that chatbot later as well.
Sure. So the first part of the question, whether we can have a live customer rep who can use this chatbot. So I have not done it, just because I was trying to build this app. And I was hoping to automate all of the questions that people have. I have not. But in Dialogflow from Google, you can also add that that third party integration, where if a question that the customer has asked does not match with any of the pre listed things that then that the chatbot can answer, then it can ping and ping you on a certain address and let you know that hey, transfer this person to a live rep. So you can do that in Dialogflow and a lot of companies here, they actually use that functionality. Besides the NLP.
Yep. And the second part was can I share the code? So for the apps, I cannot share the code just because it belongs to the company now. But I have a lot of different other Shiny apps, which use very similar methodology. You can find it if you look on if you go on my GitHub, Shatrunjay on GitHub, you will find all my apps, you'll find the different Tableau dashboards that I built some of the Flask apps that I built as well. Feel free to message me for anything that you're looking for. I'll share my email address at the end. And if you're looking for some collaboration or consultation, I would love to.
Would you mind dropping that GitHub into the chat here? I'll put it in right now for you. Sure.
The EQUAL bias-removal tool
Alright, so equal, which is which stands for eliminate quantitative bias and algorithms. This is an app that I built for an internal competition. And what it does is it tries to remove bias from predictive models. At CVS, it's it's now our number one priority. Make sure none of the models have any sort of bias in them and all our models go through this check and we make sure that all the models are fair, equal to everything.
And to do this, what we do, what the steps that we take is first, when you come in, you answer these three questions, which help us find out what are the metrics that you should be looking for fairness. So fairness, you if you've heard of fairness metrics, there are things like this part, it impact parity and so forth on. And answering these three simple questions will help you identify what metrics are important for me. This will pull all the data on on the members you're interested in. So there are 11 protected attributes here in the US. You cannot be discriminated on age, gender, national origin, religion, handicaps, handicap status, status, pregnancy, medical and behavioral health conditions. So it'll pull all of this data and then it'll check all the different models we can build. Predictive models can be regression or classification. It will check all the different steps pre processing during the model build and post processing for bias and then it will save your results at a specific place. It will give you an HTML or a Word document. And again, equal is built as a Shiny app with JavaScript, so it is zero coding language agnostic. It can work with both R or Python models that have been uploaded.
answering these three simple questions will help you identify what metrics are important for me.
So so equal was built to simplify this this pathway that the data scientist has to take from starting from the model build all the way up to model deployment. So let me quickly show you. How equal looks?
Alright, while this loads, maybe I can just switch to the last slide, which is again. Thank you and I'll take any questions you have. And again, if you are interested in working in data science for us to email me at my email, which is sing s 22 at etna.com. And if you have any other questions on code or any suggestions for the apps that you're trying to build or dashboards you're trying to build, or if you want to collaborate on something, please email me on my personal email, which is just my first name. At gmail.com.
Alright, so this is how equal looks. You can come in. You can take again a very quick tour on the different parts in in in the app and what what do they do? You can explore your data. You can evaluate the bias in your data set. You can find out is my input data biased or my models biased or not. And then you can also eliminate this bias. It uses techniques in pre processing, post processing and model build things like adversarial debiasing to remove bias from your data set. Follow some of the same steps that I've shown previously where you can explore your data. You can customize your metrics. You can find different different bias metrics. You can mitigate them and you can download all your results as well.
So I'll stop there and and take some of the questions that we have. Awesome, thank you so much day that that was great and so cool to see all the applications too. I know people are right now is still putting in some questions in Slido and if anything goes unanswered in the time we have today, I can definitely send those over to Jade and get the questions answered too. But the top question so far that had been upvoted a few times was what kind of process do you use to What kind of process do you use for updating the application? Do you have any sort of defined deployment process?
Deployment process
So the deployment process that we use is through RStudio Connect. We have a broad version, the production version and a dev version. The process that we use is first we build an MVP, a very simple version of the app which just does the basic end to end analysis and then we upload it to the development server. We give it to a small number of people and we perform test and learn so we would ask them to use it in a way that they would traditionally use it and try to and give us hints on where they got stuck and what can we improve. And this usually takes one month and then we get all the feedback from them. We try to put in the most critical elements into the dev two in the better two and then we launch it to a slightly different bigger population. And if there are no big hiccups, then we move this from the dev version to the broad version where we launch it throughout the company.
We recalibrate all our analysis every one and a half months early on and after the first six months we use a six month iterative calibration cycle. So every six months we would revisit it, see how the data changed, how the analysis, sometimes the packages are changed, is the app working as it was intended to work in the original version or not. And I've there are then we have some downtime, we optimize the app and put it back again. So that's the cycle, traditional cycle that we use.
Setting up the Shiny app tour
Great, thanks Jay. Another one of the questions that was upvoted is how do you actually quickly set up that tour of the Shiny app? I think it was in the first Shiny app that you showed. So that's through a package called ShinyBS. You can use tooltips and these introduction videos. There's another package, I think it's called IntroJS. So IntroJS was also, it's a JavaScript library and now a lot of these JavaScript libraries you will find are packages that you can, you don't have to write JavaScript code, they will add it to your UI. And so you can Google things like ShinyBS and IntroJS, our package, and you will find the packages that you can use to set up the quick tour functionality.
Shiny vs Tableau
Another question, Jay, was how do you position Shiny over Tableau? Like what would you use Tableau for and when do you decide that it would have to be in Shiny?
So actually, that's a great question. And just a little background. So I was a big Tableau user. I actually won the Tableau Massachusetts competition three years ago. So I was a big Tableau user, but Tableau has limitations. The one limitation is that you cannot do advanced analytics in Tableau. You can do visualizations, you can slice and dice data. The most advanced data science you can do in Tableau right now is you can do regression. So you can fit a simple line to your data set. And that is the limit of what you can do. But if you want to use some of the algorithms, for example, if you want to use the segmentation or the comorbidity association rules that I've used here, then Tableau cannot do that. It can make your data look pretty, but it cannot do that analysis. And to do that analysis, you will have to use either R or Python Flask.
Within R and Python, I feel like most of the research community, people like me who were in research, we tend to publish all of our research as R packages. So you would usually see things coming out of research going into R first. And if they're very useful for industry, there'll be a Python package that is now copying that R package. So with Shiny or Flask, you can use the latest and greatest data science advanced analytics, along with visualization, the visualization might not look as neat as a Tableau, just because it's been designed by like professionals, but it will still be more advanced, you can do these analysis, which are more complicated in R.
So with Shiny or Flask, you can use the latest and greatest data science advanced analytics, along with visualization, the visualization might not look as neat as a Tableau, just because it's been designed by like professionals, but it will still be more advanced, you can do these analysis, which are more complicated in R.
Clustering on big data sets in Shiny
Thank you, Jay, that's really helpful. I know, you know, that question seems to come up on quite a few meetups, too. So maybe it'd be helpful for us to even have a full a longer talk on that, too. Another question, I know, we have just a few more minutes here. But should age asked, Hey, Jay, have you encountered scenarios where you want to run clustering on big data sets, upwards of 10 million rows from within Shiny?
So, so the one thing I would say is that most of the Shiny apps that I've shown you, it tends to run on smaller data sets. For bigger data sets, Shiny for for the current for the knowledge that the limited knowledge that I have in the use that I have had, the scalability tends to be an issue later on. Now, recently, there have been new packages that you can use to paralyze the analysis. And you can use delay methodology. So you can, you don't have to perform the analysis. Right, then you can use a catch or you can delay the analysis later on. And that makes the flow, the user experience faster. But if you are trying to use big data analysis, then Shiny might not be the best idea shortage. Or it might be maybe I don't know. But for those kind of analysis, trying to build a live app, you tend to work better with some of the other big data methodologies. So if you're running your analysis on, say, Spark, then you can use things that that go on top of a Spark analysis. And you can use dashboarding there to analyze those process flows, and they'll be much more efficient. And they won't bug your users than a slower app, which tends to bug people a lot.
Bias detection app resources
Thank you. And for one of the last questions today, I see Tony asked, could you talk about the bias detection app? Is there any public info on that?
So there is a lot of free public information out there. IBM came up with something called IBM 360, Fairness 360, which is a comprehensive set of tools built by people who are actually researchers and professors at university. Very extensive package. It's both an R and a Python package, AI360, which for me, the one thing with that package is it tends to be confusing just because they put in so many different details. For a user who just wants to check his model for bias, I would be a little confused on what metrics should I use? And what does actually bias mean? So Google has come out with their own fairness package. It's a web app that you can quickly use, import your data set and find out whether, say, there is race or gender bias in your data set or not. So there are different methodologies that are out there. For your own specific customized use case, I would say you would have to actually study the material and then come up with what is the most useful for you. But a lot of literature out there.
Great. Thank you so much. I know we're just getting to the end of our time that we had scheduled today. So I just want to say thank you so much, Jay, for the awesome presentation.