Resources
Feb 7, 2023 |
1h 3min
|
3.7k views
|

Posit Academy Overview

Over the past few months, we've featured a number of stories from amazing pharmaceutical leaders at Roche, AstraZeneca, GSK, Eli Lilly, Pfizer, and Janssen - sharing their perspectives on the adoption of open source. You can read more about that here: https://posit.co/blog/open-source-in-pharma-from-five-perspectives/

One of the common questions we've seen across many of these meetups, webinars, and data science hangouts is how teams are training the next group of users and questions about best practices for those learning.

This shift to open-source data science requires many statistical programmers trained in legacy software to learn open-source languages like R and Python.

On Tuesday, February 7th, at 12pm ET, we will be joined by Garrett Grolemund, Director of Learning at Posit, to share how Posit Academy teaches programmers in the pharmaceutical space how to use open-source tools within the context of clinical data.

You’ll walk away from the presentation having learned how Posit Academy is different from traditional learning programs and how some of the largest pharmaceutical companies are using Academy to upskill their programmers in open-source data science.

Presenter:
Garrett Grolemund, Director of Learning at Posit Academy and an award-winning R instructor. He's the author of Hands-On Programming with R and a co-author of the best seller, R for Data Science.

To learn more: posit.co/academy
Link to slides: https://github.com/RStudioEnterpriseMeetup/Presentations/blob/main/Posit-Academy_Pharma-Meetup.pdf

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Happy Tuesday. Hope you're having a great day. Welcome to today's Posit Enterprise Community Meetup.

Over the past few months, we've featured a number of stories from amazing pharmaceutical leaders at Roche, AstraZeneca, GSK, Eli Lilly, Pfizer, Janssen, all sharing their perspectives on the adoption of open source.

One of the common questions that I've seen across many of these meetups, webinars and data science hangouts is how teams are training the next group of users and questions about best practices for those learning.

I'm so happy to have Garrett Grollemund, our Director of Learning at Posit joining us for today's presentation. Garrett is an award winning R instructor, the author of Hands-On Programming with R and co-author of the bestseller, R for Data Science.

I also wanted to say thank you to Devin Pastor, one of our principal solutions engineers who works with a lot of our pharma customers at Posit, who is helping out in the background will also join us for the Q&A. You may see Devin answering a few questions in the YouTube chat as well.

So with that, you can use YouTube for questions, but you can also ask anonymously through the Slido link that I'll pull up on the screen here as well. And we'll copy that over into the chat too. But with that, thank you all so much for deciding to spend some time with us. And I will pull Garrett up on stage here with me and turn it over to you.

All right. Thank you, Rachel. Can people see what we want them to see? Looks great. I can see that you can see your title presentation and I'm going to jump backstage.

Well, hello, everybody. As Rachel said, my name is Garrett. I am the director of learning at Posit, but I've been working with Posit, formerly known as RStudio, for about a decade, mostly teaching people how to use R, to write code, to do data science, and the other things that you do with R. And now more and more Python too.

Many of the clients that I've worked with were people who were trying to migrate from SAS to R or from Excel to R or you might just say nothing to R. So they're trying to train teens to learn how to code, do reproducible work, and they do program-assisted data science. What I discovered is you can teach adults, no matter any age, to do this very effectively. But whether or not the training ends up being a success depends on how you do it. And that's going to be the theme of the talk today.

How should you train your colleagues to adopt R or Python or any code language to do what they're formerly doing in SAS, Excel, or on their own?

The limits of workshops and online courses

Now for a lot of those 10 years I spent training people, I was teaching workshops. I'm going to try to convince you today that workshops, standard training of all varieties, really is not an effective way to retrain or retool a workforce to use code. It's also true with online courses. But I'm going to show you how you have options beyond these things to do a very good job of it.

So how do you train a data scientist? First off, I want to start with my story. When I joined RStudio 10 years ago, it was a very different world. A lot of people didn't know about R, or if they did, they had these impressions of R that were not very complimentary. So we had to spread the word about R, and we decided to do that by teaching people to use R in the tidyverse, which we were creating at workshops.

We also experimented with other partners to do video courses, online courses, MOOCs. If there's a way to teach data science, I have done that. And if you think of someone who teaches data science, there's a good chance I might even work with them to help them do that as a partnership.

And the reason there's nothing on this slide right now is because I want to talk about some of the negative experiences, and I don't want to associate that with anyone in particular. But what I quickly noticed teaching data science was that teachers out there really aren't that good. The workshops you see at conferences are not so great. And even some of the online courses, I really questioned how effective they would be.

And after thinking about this, I realized the people who become experts at coding or data science did that by devoting years of their life to becoming that expert. But they didn't have a chance to devote those years to being a good teacher, or even find out what made teaching work.

Serendipitously, my background is in psychology. I have a PhD in statistics, but that's from doing psychological research, I work my way over to writing code that supports statistics, and now what we call data science. And my background in psychology alerted me to the fact that there's many things that you can do to make training more successful, and I thought, well, there's these theories that are well established, like cognitive load theory, multimedia learning theory, all the rest. They all apply to learning. And if we apply these to the data science training environment, we should be able to make a much better workshop.

If there are any trainers in the audience, I strongly recommend learning about these things, and there's a book that makes it really simple to learn about the research. All the most important educational research is covered in this book here. It's called How Learning Happens by Paul Kirshner and Carl Hendrick. So that's something you might consider if you're developing your own training programs.

Anyways, I took this and I put it into my workshops and my training programs, and it worked amazingly. People love the workshops that I was creating. I started to win awards like crazy.

Correspondence was sent to me from the American Statistical Association because I'd come to teach at their JSM conferences, and they'd poll all the students, their statisticians, and the workshop that was highest rated amongst many, many workshops would always win an award, the Excellence in Continuing Education Award. Well, I won it three years in a row, and that was unprecedented, and then because of that, they hired me to go around the country and teach at different chapters, including Hawaii, so I got to go to Hawaii as a teacher. The picture of me jet lagged over their teaching. Students gave me LA. That was pretty cool.

But I thought, wow, this is great. I've cracked the code. I can teach a really effective workshop. These students are telling me in every workshop that learning R is going to be life-changing for them because they could see how it would be so much easier to do the things they're doing and they'll actually be able to do some more. It'll be like a phase shift because they have the reproducibility and executability of their code.

So I thought, the world looks pretty good, but I still have a background in psychology and I started exploring educational psychology, the literature of psychology that's related to education, and there are some disturbing signals in there, and I'll share some of those with you.

For example, the best we could tell based on research is that students only remember about 56%, so about half of the content, that you tell them in a lecture. And that's if you test them immediately after the lecture. And we expect that this drops off from there as time goes by. There's the study where I'm getting this number from. But this is sometimes very disturbing if you're a trainer. Not only is this disturbing, the students who are being tested here are undergraduates at college who we might think of as professional learners. If they can only manage to remember half of what they learned in a lecture, then the distracted adult let's go right back to all their other work responsibilities, probably go learn even less.

That's not very good. Next other studies, here's a study that looks at procedural instruction. So this is more like teaching someone how to do a procedure, whether that's using a machine or fitting a certain model or something that you do, it's a behavior. So it's not like something you take a test on later. And they find the same numbers immediately after the training, where the students are doing the thing, only 60% of them can reproduce it. And then six months later, that drops to 40%. One year later, that drops to 30% and so on. So that's not great either.

And most trainers I discovered weren't aware of this at all. But you can look at the students being trained and kind of get the sense that this is exactly what is happening.

I went back to my workshops, and I continued to teach, and I continued to get rave reviews. And I realized what all the knobs were to make student satisfaction really high in a workshop. But I also realized they don't really relate to the stickiness of what the student's learning. And anecdotally, I started to check back in on some of my students, and I realized, even though they love the workshop, and they could do lots of great things in the workshop, they weren't doing those things a year later or six months later. To me, that seems like a training failure.

And I'm not going to speak for everyone, but I almost feel like it's a dirty secret of the training industry that you don't get terrific outcomes as a trainer. I've heard people talk about marketing and say, you know, we know half of what we do works. We just don't know which half. Well, it seems like that might be the case for training as well. But we're dealing with a field where people aren't trained to be trainers. So I think there's lots of upside to do much, much better. And that's what we set out to do at RStudio, which is now Posit.

And I'm not going to speak for everyone, but I almost feel like it's a dirty secret of the training industry that you don't get terrific outcomes as a trainer.

The forgetting curve and how the brain learns

And the way we fix this is by looking into why this happens. So some of you may have seen this graph before. It's called the forgetting curves or the ebbing house curves based on the gentleman who first postulated it. This summarizes a body of research that goes all the way back to the 1800s, has been verified in many different fields. It's just a fact of human life.

The way to look at the graph is these blue lines are someone's ability to do something, be that to do a task, to remember vocabulary, to take a test. It's just their ability to do that thing that they learned. And the first blue line shows how that ability changes over time. Depending on the ability and what the training is like, it drops off at different rates. So there's no units on that axis. This is more of a model here. But what happens consistently is your ability to do something drops off after you first learn it. And especially if you never use it again, your brain basically cannibalizes the resources that devoted that ability and uses them to do other more important things. And eventually you can't do that thing anymore.

But you don't have to settle for that outcome. If you wait until that thing starts to get difficult for you, and then you practice it and do it again, you're essentially relearning it through your practice. And you do it again, and you jump up to the next blue line. So your ability started to fall off, but you practice, and now you can do it again pretty well. And then it starts to fall off again. But it doesn't fall off as fast as it did the first time if you had never practiced it. And if you repeat this process and you practice again, you'll be on the next blue line, and then the next blue line.

Whenever you don't use something, your brain does start to lose its ability to do it. But the more you practice the thing, the more your brain decides to conserve those neural networks and retain that ability. Now if you've practiced something enough times, you've learned to ride the bike and it stays with you forever.

You could sort of see why this doesn't play out in a workshop that you take for half a day or two days or what have you, and then you never really see it again. These practice sessions work because your brain is building new neural circuitry in between. That means you have to go to sleep, nighttime sleep, in between these sessions to really build up a robust neural network. I have occasionally had people sleep in my workshops, but I don't think that was causing the effect we look for here.

I'm also realizing I could have used these slides to explain this a little more clearly, but this is what I just said, how the more you practice, the more it changes the trajectory of forgetting until eventually you don't really forget.

All right, so we could see in the data that's how that works, but we also know from research and study why it happens. This is your brain. It's a collection of neurons that are joined together, and your abilities and the information you know and the memories you have are all encoded into your brain as networks between these neurons.

The network is set up so when one neuron in the network fires, the other neurons fire as well, and they trigger whatever behavior needs to come together to make this experience that your brain is recreating for you. We can imagine having two networks that do two separate things in our brain. One of these we use, every time we use it, those neurons fire together, and then your brain wires them together a little more strongly, because this is how the brain works. You probably heard neurons that fire together wire together. Every time you use it, that network gets stronger and stronger, whereas the one you don't use gets weaker and weaker.

Your brain has important things to do. It's not going to maintain a network that doesn't do anything useful until eventually over time you'll have a very, very strong network at the thing you were doing, so strong that doing it no longer takes conscious thought. It's completely automated. When one of the neurons in the network fires due to some cue or trigger, all the rest just automatically cascade, and you juggle the ball, or fit the model, or interpret the data. Whereas that other neural network that we never use, it doesn't even exist anymore. It's gone.

This is the biological phenomenon that explains the forgetting curves. Like I said, this wiring together occurs while you're sleeping, so this is something that happens with lots of repetition over a long period of time, weeks, let's say.

If we go back to our model and think about, do we want our colleague to be able to do this new skill, pick a skill where the answer is yes, and then consider, what is going to determine whether or not he or she could do that six months from now? You'll discover it's very little to do with the initial instruction we give the colleague and everything to do with the amount of practice they've had shortly after that instruction.

You'll discover it's very little to do with the initial instruction we give the colleague and everything to do with the amount of practice they've had shortly after that instruction.

How do we train to take advantage of this? How do we train to make people who can use R as well as SAS six months from now?

Something else to consider as well, just watching a lecture or watching a video on your computer about doing something probably builds a neural network, but it's a neural network that's related to sitting still and watching a video unfold. It's actually doing the thing that builds a neural network so you'll later use to do the thing.

We have to be conscious of that in training too, and unfortunately a lot of training doesn't really focus on the student actually doing the things. It's more, hey, watch this video. Maybe there's a multiple choice question afterwards, maybe there's not.

Data science as a skill

What does this mean for us? Well, the revelation we had at our studio is that data science is a skill. If you want to learn to do good data science with code or otherwise, you have to practice it and learn it as a skill. A great analogy would be playing the piano or some musical instrument. Those are skills as well. There are things you need to learn like how to read notation, rhythm, music, but you also need to sit down and practice the piano if you want to play a song on the piano. You wouldn't expect to do this well by watching an online course or videos online.

Maybe I can supplement what you are going to do to learn to play the piano. You wouldn't expect to learn to play the piano well either by attending a short workshop that lasts for a day or two days. I mean, it might help, it might get you enthusiastic to do the practice you need afterwards, but at the end of the day, if you want to play the piano, you do need to practice over time. And usually with some sort of feedback from an expert who's mentoring you as you play the piano. And the same is true for data science. We need to use practice over time to learn to do it in a new way.

The problem with online courses

All right. So what about online courses? They can last over time, right? Well, what we find or what you would find too, if you research the training communities, students don't complete online courses. They don't stay very engaged with them even when they do complete them. And that being someone who's designed many online courses, I can tell you, it's very hard to design an online course that multiple students can come to and be challenged by without a teacher there overseeing the process. You sort of have to design for the lowest common denominator, and that's not enough to help the rest of the class learn.

When we look at really well-supported massive online courses, like the Johns Hopkins or Coursera MOOCs, we see that it's typical for about 10% of students to finish those. I've also had the privilege in my life to look at the data for some online data science training providers because we were partnering with them, and what we saw was something that was very similar to a gym. Lots of people pay for a subscription to go to the gym, but at the end of the day, they never show up and go to the gym. They're just sort of paying. It's great if you're a gym and you're collecting money, but it's not great if you're paying to get student outcomes because if the students don't go there and engage, they don't change.

And the missing element for these online courses, whether it's a MOOC or this gym model, is motivation. So lots of people think education works this way. There's an educator. They have the information that the student needs. There's a student. We just need to transfer that information over, and then the student's now an expert.

But that's not how education works at all. Information is cheap and easy, and you could get it from the library or YouTube or wherever you want it, but if you want to learn to use that information, you have to do something after you acquire it, and this is the hardest part of learning something. You have to practice it in a loop over and over again, and it's easy to start practicing things the wrong way, and as a student, you might not even know you're doing it, and then you're just building neural networks that encode mistakes. It's not going to work out well. You need to practice and get feedback, maybe some coaching.

Practice itself is hard work, and all of that work requires you to be motivated. It's hard for busy adults to stay motivated at one more form of busyness, but this blue loop here is what matters if we want our colleagues to become excellent R users or Python users or data scientists. Most training environments ignore the blue loop. It happens after the training's over. So what we decided to do is focus on that blue loop to make it easy to do the hard work of learning, to do it as efficiently as possible so you know it's paying off, and to give you the coaching that keeps that practice loop where it should be so you're practicing the right things instead of spinning off into mistakes.

Practice itself is hard work, and all of that work requires you to be motivated. It's hard for busy adults to stay motivated at one more form of busyness, but this blue loop here is what matters if we want our colleagues to become excellent R users or Python users or data scientists.

How humans actually learn skills

So what does that look like? How do humans actually learn skills? Might seem like the training industry is botched, and if you want to help your people transition over to R, you have a hard time of it, but it's not actually the case. We learn skills every day in modalities that are very familiar and that lots of people have confidence in. So we use pianos as a metaphor earlier. If you wanted to learn to play the piano, you would find a piano coach or music teacher and they'd give you some one-on-one feedback, and it'd be a process. But every apprenticeship also works this way, every internship, every residency. If the thing you're trying to learn happens to be a physical skill, we call it coaching. These things work all the time. We need an experience like this for using R, for doing data science.

The Posit Academy experience

So how should your analysts learn data science or code or R or Python? We've put together an experience that looks like this, and this is my answer. This is how it should look. If you train with Posit Academy, the first thing you have is a mentor. A mentor is someone who knows how to do data science with R, and they're going to help you. They're going to coach you to learn it. We have two types of mentor at Academy. One is the type that we provide. We provide experts to help people learn, and the other type is a mentor that the company who engages us provides. They act as an apprentice in the course experience, an apprentice mentor. But they're another resource for students, and the cool thing about apprentice mentors from the company is after the experience is over, that apprentice is still at the company, still acting as a mentor, still forming the node of a support network within the company, so you build up a community of practice.

The next thing you have in addition to your mentor is group mates. We've designed a cohort-based learning experience where no one learns in isolation. They learn in small groups of five to seven students who are all trying to learn the same thing as you are. So as you go through the course, not only do you have an expert who has your back, who's coaching you, you also have fellow travelers who you could discuss things with. You could work through problems together, and you may not even realize it, but you can hold each other accountable and motivated as you go through the process.

So those are the people involved. Next we give you a project to do. So an Academy course is not a course. It's an apprenticeship. You're going to do something very real and very close to what you do at your job because that's ultimately what you want to learn how to do. We want you to see the value of everything you're learning, and the best way to do that is to make it similar to your work, which is very valuable to you. So an example project would be comparing different treatments, maybe with asthma, maybe with some sort of disease. If you came to us from finance, we'd give you a finance project and whatnot, but we give you something that is close to what you're going to do, and you're going to do it under the guidance of the mentor and with the support of your group mates.

Each project is scoped to cover what we think or we have found over the past decade is essential to learning R. So that begins with importing data, exploring the data with plots and visualizations, wrangling your data, analyzing it. If you're familiar with R, I'm talking about dplyr, tidyr, joins, that sort of thing, but also fitting models to your data. And then how do you summarize your work to communicate with someone else, and how do you do that reproducibly so the next time you have a similar project, you're already way ahead and you start with your previous work versus making it up from scratch. We focus on R Markdown and now increasingly Quarto, which is the next version of R Markdown.

So you're going to do all of that to complete your project and do that real life thing that you will presumably now be able to do at your job and create a lot of value with. But how do we make that happen? It can't happen out of thin air. We break the project into 10 progressive milestones. Each week you work on one of these milestones. When you get to the 10th one, you finish the project, you have your portfolio piece, which is a report that shows the conclusion of all your data science work. But each of those milestones build up to that in sequence. So first you learn the basics of working with R, how to troubleshoot, what to do if you get stuck and you need help, or what to do if you hear of a new package or a new function and you want to teach yourself how to use it. We teach you how to teach yourself. That's a big part of working with R. Then you learn visualization, then you learn data wrinkling and joints, and so on.

Think of the project as the skeleton or the backbone of the course. Most courses have a syllabus, but we have a project, and it functions a little bit like a syllabus. You're going to need to do all these real-life things to finish the project. We divide into our milestones, and then we prepare tutorials that teach you what you need to do to do that milestone. So if you have to fit a model in R and you're just learning R, you don't know how to do that. We're going to assign you tutorials from the library tutorials we've written that will show you how to fit models in R. These tutorials are designed to teach through exercises. We've written grading code to give us a type of artificial intelligence that will look at the code you submit, and if it's wrong, they'll give you advice on how to fix that code. It doesn't focus on unit tests or making sure your result is right, it looks at what you're writing and tells you how to change that, because that's what you need to learn.

And then at the end of that week, after you've learned your tutorials, done your milestone, we have you get together with your group and present what you've done. We ask you for each milestone to recreate something, a step in your project. If you've studied the tutorials, you should be able to do that recreation. But then we ask you to teach yourself something new to create a personalized or extended milestone that you could present to your groupmates. This creates a lot of cross-pollination when the groupmates get together, they learn from each other, but it also helps you practice teaching yourself new parts of an ever-expanding free and open-source language like R.

The group sessions, we hold them over Zoom or Microsoft Teams, looks something like this. So some of our former illustrious clients. You share your screen, you're going to be working with the RStudio IDE, which is a real tool many day scientists use. What's in Posit Workbench, you also end up working with Posit Connect, so you learn how to use the real tools that you can start using the day after academy ends. In fact, most students start using them well before academy ends at their work to create value.

Your groupmates give you the accountability. You have to show up with something to show them. You also meet with them during the week one time for a co-working session to work through trouble together. But they're more than just the people who make sure you did your assignment. After the course is over, they're your future support network. They're the beginnings of a community of practice that you've established by learning R inside your pharmaceutical company.

So a typical week, do some tutorials, complete and personalize a milestone, present to the group. In between, you have a co-working session with your groupmates to help out. It's optional, but we strongly encourage people to go, and most people do go. And then you have the communication channel like Slack or Microsoft Teams where you can discuss the project and answer each other's questions. And your mentors are there too to answer questions. Your mentors also have each of the in-person sessions. Repeat this until you finish your project, and that's what the academy experience looks like.

Results and student outcomes

So does this work? Can this teach someone who doesn't know R to use R and do real data science with it? The answer is yes. We've taught about 600 students so far using this method. We have about a 95% completion rate. This is for a course that lasts 10 weeks, 12 weeks normally, because we put in some breaks for holidays and stuff just based on how the year works out, and the students stick with it.

Here's an example student, Keith Wickle from Kellogg Company. I'm calling him out as an example because he posted on LinkedIn in public. But what we see with other students is similar to what we see with Keith. People are really excited to be students in this sort of experience. They're really excited to actually apprentice on something, learn it well, and have a mentor there to help them through any question that comes up, any challenge that they have, or anything they want to explore that we didn't put in the course, but they know they're interested in. They can do it and discuss it with their mentor and group mates.

So our students like it, but our customers are actually companies. We're trying to help companies create capacity to do work that didn't exist previously in their company. And it works there, too. This is probably the most on-the-nose quote from a previous customer. James Wade shared this at a previous RStudio Enterprise Meetup. I guess now there might be positive Enterprise Meetups. And he said, they were very interested in whether this was working, so they went back and they surveyed the students they put through Academy six months after they left Academy. And they found that 16 out of the 17 students were still writing code at least once per month or more frequently, using what they learned in Academy. And that 17th person, by the way, was promoted to a manager, so they no longer had to write code, but they were convincing their direct reports to start using R and write code based on what they did at Academy.

And they found that 16 out of the 17 students were still writing code at least once per month or more frequently, using what they learned in Academy. And that 17th person, by the way, was promoted to a manager, so they no longer had to write code, but they were convincing their direct reports to start using R and write code based on what they did at Academy.

Now I don't know how you feel about that, because you're not a trainer, but perhaps based on what you saw at the beginning of my talk, that this is unheard of in the training community, 16 out of 17 survey response. But it's not unheard of out of our customers. This is par for the course, but James went on record about it. We have taught pharmaceutical companies.

There's a quote on our website, paws.co.uk slash Academy, where AstraZeneca talks about how great their Academy experience was. It's more general language. I'm a data scientist. I chose this quote with data, but it works. People notice the difference and they sign back up for more cohorts of more students.

Anyways, thank you for listening to me. You may have noticed I'm not a pharmacist. I'm an educator, and I geeked out on education, but I hope after hearing this, you get a sense of what you need to do to successfully transition your colleagues to R or to Python from wherever they're coming from. These principles work in every circumstance, and if you avoid them, you're going to have fleeting gains at best.

Thank you, everyone. I'll stop screen sharing now.

Thank you so much, Garrett. I'll pull your screen backstage here. I see a few questions coming in now, but I also thought it might be helpful. I know you were going to maybe pull up Academy and show one of the examples in there, too. I thought it might help for a frame of reference for some of the questions, too.

Sure. Shall we start there? Sure. That would be great. All right. I will share my screen. All right. There you go. If you want to just make that one full screen and then pull it over, there we go. Like that? Yep. I can see it now. All right.

Tour of the Academy student experience

This is the student landing page or an example student landing page for the Academy experience. Now, remember, so much of the experience is about that social motivation you get from your group and your mentor and talking to them outside of this platform here. But when it comes time to do your learning in a tutorial, this is where you go. We have a syllabus on the side that's divided into different milestones. You can see first we're just learning R, then we're importing and visualizing our data, adding variables. These things are progressive. They build on each other until eventually you're getting models and writing your own reports and publishing them.

Within each week, we have the lessons that you take and then the milestone that you do. You go into the milestone file. It takes a minute to open up, but it's opening up the RStudio IDE. We provision the IDE with the different files that you would need or that you might find at your job if you were doing this type of task. And we have an R Markdown document that guides you through the task. This is the first milestone. It's kind of like a hello world in some ways, but we gave you some data, we want you to open up, we want you to look at it, we want you to recreate something.

In this case, we give them an image of a plot and we say, there's a function that does this. It's the hist function. We did not tell you about it this week, but we did tell you how to read help pages and learn about functions. So can you learn about this function and make this plot? And then we give them an extension, which is, okay, let's push this a little further. Maybe you could change the color, maybe you could scour that help page and learn about your options and try to apply it. This is the first milestone. They get more and more advanced as you become more comfortable with R.

Q&A: when does creative problem solving begin?

I was going to jump over to a few of the questions too, and we can mix it in and maybe show through Academy here. But I see Ben had asked, thanks, Ben, for the question over on YouTube. At what point in the process would the ability to creatively solve problems be expected? Curious where the complexity of practice should be introduced.

We don't discourage creativity for solving problems at any point. As the milestones get more complicated, they sort of become more divergent. So in the second week already, you'll be making plots and each project is guided by a motivating question, which might be as simple as, you know, is there a difference between these two treatments? Or it might be, you know, what factors predict this thing? And normally within those questions, there's multiple relationships you might search for or discover, especially if it's what factors cause this, who knows?

So we encourage the students to be curious, explore as many relationships and data, maybe even make their own new variables or features to explore and visualize with different types of plots. Even when you're comparing treatment groups, you might want to look for differences beyond, you know, the outcome of that disease. There might be meaningful differences in like blood pressure or things that we'd want to know about.

So I would say as soon as the second week, there's lots of room for experimentation, exploration, reporting new things back. Also that extension piece allows people to really learn a lot of different things. For example, I've had a student who thought, I like the plot I made, but this would look really interesting animated because there's something happening over time. Ours free, it's all free and it's all very accessible online. So he went out and taught himself the GG animate package and brought that back. He was able to do that because we talked about how to learn new packages. So I would like to say that this is built into the process.

Now what happens that isn't apparent from anything I could show you here is the students come into a session and they're talking about doing an ANOVA or whatnot. And maybe the data is heavily censored, right? And they have a mentor and normally a company mentor as well. And all the ingredients are there to have a discussion about what is the best practices for this type of data? Like, how do I go about solving this problem and thinking about this problem from the apprentice mentor or the mentor side, a relationship develops within the group that goes to the places where the group or the mentors think is useful.

The MDRD project walkthrough

Something else I wanted to make sure that we cover, I mean, especially given the title too, but I know you have an awesome example in the Academy project gallery that covers modification of diet and renal disease. I thought that might be a great example to share.

You all can see my screen, right? I can't see it yet. I hit it backstage, but I actually just grabbed the link for you too, so that you had it in the private chat. Okay. Well, I can see it. Oh, I can see it. Okay. Let me pull it over. There we are.

So when a new customer comes and talks to Academy, we try to find the project that fits best with them. So we have many projects that already exist and we've developed a consultation with clients and whatnot. And one of them is, no, actually that's the wrong one, but you had a specific one and that was- I thought because it's focusing more on clinical, there, right there. Yeah. Modification of diet and renal disease.

So this, sometimes we have to be clever about getting around HIPAA and regulations and stuff. So this is data that's actually been published in scientific literature. So it's not going to trouble for using it. And it is a clinical trial that looked at four different treatment groups on renal disease, so kidney disease, and the treatment groups varied the level of blood pressure medication the patients took and also the amount of protein in their diet. So we had four different groups and ultimately we run an ANOVA to determine whether or not there's a difference between the groups.

These are the 10 milestones in the project. So first they're going to create a histogram of the measure that we're interested in, how much your kidneys can filter, the measure of kidney health. We progressively get them from using Basar. So first we just want to make sure they could use our functions. Then we start teaching packages like ggplot2, that's the graphics package in R. And they group and summarize by patient to reveal things that you wouldn't find in the data as it is, but you can derive from the data. We join in a second data set that they can work with. We tidy and pivot tables.

We start reporting our results in a way that makes it useful to turn in assignments, but also to turn in reports or decisions to people that your team is working for. We work with different types of data such as strings, dates, times, depends on what sort of data you would work with at work. That's what you'll work on on your project, but we do teach you how to work with each type of data in the tutorials. And then here, we have some divergence between fitting models and writing functions on data.

Here's some of the artifacts students make over the course of 10 weeks on this project. The way this project looks to the students, though, this is sort of an executive summary that I could share with my colleagues. What the students see is what I was showing on Posit Academy. This is the actual MDRD project. So we get to Milestone 6, MDRD stands for Modification of Diet and Renal Disease. You see we're talking about globular filtration rate. This is the project.

In week 6, they get this milestone. Run the code below to make the table. This is not the code you'd write to make the table. It's just code that delivers off the table we made. So now they have to figure out how to take their data set, which they're very familiar with, and fit a model that returns this table here. And then they can check to see if they did it right, if it looks the same. And then they can extend that, changing maybe the variables in the model or whatnot.

Q&A: how is Academy structured for life sciences?

Thank you for walking through that. I see a lot of questions that are coming through from both Slido and on YouTube. I'm going to pull one over from Slido here. And I can hide the screen so that you can see it on there as well. But one was, can you share more details on how Academy was structured for customers in life sciences? Or are there any measures of student engagement and overall success in life science?

Yes. So, first of all, Academy is not structured specifically for life sciences. It's structured specifically for people who need to learn R or Python to do what they do. We've become experts in teaching R and Python. And what we found is it's not very different whether you need to use R to do this or to do that. It's not even that different if you're coming from SAS or Excel. If you're coming from SAS, we can use the knowledge you have about SAS as a springboard to connect to R. But at the end of the day, we're not going to give you a translation guide that shows you how to ask where the bathroom is in R. We want to teach you to teach, to speak fluently in R, and that's what we achieved in 10 weeks.

We're not going to give you a translation guide that shows you how to ask where the bathroom is in R. We want to teach you to teach, to speak fluently in R, and that's what we achieved in 10 weeks.

So our structure looks the same for life sciences as for other industries. You need to know how the mental models of the R language, so it's divided into functions and objects, and this is what those mean. You need to know where to get your help from, whether or not you're going to use clinical trials data or preclinical trials data, the R language operates the same way. You need to know the tools that execute your code. You need to know procedures for backing out of something that didn't work and finding the right path forward.

Then there's a lot with data that is also remarkably similar across different fields. So we make sure you know how to get your data into R where you could use it. If you have really big data, we'll teach you how to get R to the data in the database using the dbplyr package. If once you have your data, you need to know how to join tables together, R works best when you put everything in one table before you analyze it, we teach you that, and that's field independent. Tidying your data, R works best if your data is in rectangular format or tidy data format, we teach you that. How you make a plot is the same.

There's so many commonalities across the industries that really, there's just one and a half or so weeks that are different from task to task at this level, and that's what the projects capture. So perhaps that's what I should be talking towards. Each project teaches the scope of all those things. It's the scope of what you'd find inside the book R for data science, by the way. But it takes the payoff step, what I call the payoff step, the model that's valuable to you with the type of data you use, and the tweaks that are idiosyncratic to the type of data you use, and it matches those to your industry.

So how do we structure things for life sciences? We made projects that looked at data that life scientists or life science industries use, and that asked questions and answered the questions that life scientists ask an answer of that data. We don't train anyone to be a life scientist, we train them to be a life scientist who uses R, but we assume we're meeting them as life scientists already. Now we just show what, how to use R to do the things that they're interested in.

Thank you so much, Garrett. And I know we had Devin backstage kind of helping with some of the questions in the chat too. So I wanted to pull you on here as well, Devin, because I know you work with so closely with many of our life sciences, life science customers as well. I just want to see if you had anything to add to that too.

Yeah, thanks, Rachel. Can everyone hear me okay? Yeah, you sound great. Awesome. I think part of the flavor that I can add that Garrett really hit home throughout the entire webinar is that to frame Academy against some of the leading questions that you'll

often see is that groups are coming with very differentiated needs, you know, even in very specific areas like just clinical programming, right? For some groups, we have this large set of SAS programmers, and we're trying to convert them to R. For other groups, they're trying to do novel analytics that they've never done before. So they're coming from like a clean slate.

But ultimately, a lot of the patterns across companies across groups are still consistent. And so part of the beauty of Academy is bridging a lot of the pedagogical techniques, but recognizing the fact that the more domain specific they become, the more they're going to immediately resonate.

And so I think, you know, with respect to like the analyzing clinical trial data elements, this is absolutely a call to say that, you know, we have been engaging with companies interested to do that, looking to develop a program that fits what type of work that they are trying to do. But it is a conversation to have, it is not a kind of a routine, hey, everyone is going to go through these same motions, and hopefully they'll come with the same outcome.

Instead, it really is about saying, what is your group trying to do? We understand and we can speak towards some of the high level objectives that most groups are coming in with, we know you're going to need to generate certain types of tables and figures and legends, or expect to do particular types of analyses. But ultimately, trying to teach that highly specifically is still not going to position a group to be able to take the learnings from Academy and extend it to all the scenarios they want.

So just infused throughout the entire program, is that desire to both empower people to like, think in terms of how they're trying to approach their analytics, and give a lot of those pedagogical techniques like repetition to make sure that you're able to successfully on ramp to the language.

And that's one of the observations that in kind of my last decade or so of interacting with groups is, even if they crack open a book, or they crack, crack open some courses, when they start to work on their own projects that they can't, you know, copy and paste some code snippet to work off of, it's easy to get frozen as soon as something doesn't work. And you don't really know where to go.

And groups that go through Academy, we see things like they're enabled and empowered to be able to troubleshoot to diagnose and to kind of move off of the path that was covered, even within the given context. And so that's why we really look forward to being able to continue to extend those teaching engagements and really are looking for people to broach what are they interested to learn.

And the answer in most cases is not going to be like, no, we won't teach that is a conversation about how are you trying to, to empower the group and what. So I think there are some of the questions of like, you know, will you teach a, you know, doing cohorts analysis or certain regression like that is, those are the questions that we're looking to start that dialogue with as part of this webinar and into the future. And so I don't know if there are any that would bubble up at the moment, but if not, I think that's the kind of question that we're looking to engage with groups to learn more about their specific needs and figure out how to make them successful.

Audience Q&A

Thank you, Devin. And I see a comment in the chat, Stephen had mentioned, I really liked the learning curve slide and would like to reuse that and asked if some of this would be available. So I wanted to make sure I shared on the screen here that you can access the slides at this short link, we can reshare that in the chat as well.

But with that question asked, I was just curious for people that are joining us here today, are you kind of in charge of training some of the new users across your team or are you part of a data science community within your company as well? If you want to put into the chat, I'd just love to learn from you all there too.

But an Academy specific question that I see from YouTube, Garrett, is how are the groupmates paired? Are they assembled by similar levels?

We've experimented with this and what we do right now is we pair groupmates based on similar objectives, like the tasks to be done, they go do similar tasks. What that means is you could have some really advanced R users who may have been using R quite a bit, but not really the tidyverse or maybe not for the tasks to go do with brand new beginners. And that works quite well. Having them in the same works as well too, they both work well.

But what's interesting is when you have people from different levels, the ones who are more experienced act as sort of mentors because you're putting them together in this team group experience. And any teacher will tell you that's a more effective way to learn than learning. And teaching is a great way to learn and they're there and they're teaching.

And since the recreations are, we feel they're doable for beginning students, no one feels like they fall far behind, but the extensions, they can run as far and as fast as they want to. So those who are ready for more of a challenge can push themselves. The mentors are there to help them no matter what stage they start to get bogged down. And those who are beginners, well, they push themselves too. It just so happens to get bogged down here, but the mentor is there to help them too. We've had a lot of success mixing models.

And any teacher will tell you that's a more effective way to learn than learning. And teaching is a great way to learn and they're there and they're teaching.

Thank you. One other flavor that I can add as well, that again, is another observation from interacting with a lot of these groups, both within Academy, not with Academy, doing workshops, is that there's often a perception from a lot of folks in the pharma sphere that the data science activities are like a small component of their job. Maybe they need to also be considering around how decision-making is performed, other more domain-specific activities, but ultimately a lot of their day-to-day is still spent in R doing work.

Even if you have been exposed to R for a while, and you might be an intermediate even pushing into advanced, there's still a lot of value in reestablishing those fundamentals in a way that you probably had to learn on the fly on the job, where some of the activities that you used to say like, well, I know the motions to fix this. I don't exactly know why, but I wave my hands for a bit, and then eventually it works. All of a sudden, a lot of those concepts suddenly become clear and become emphasized and covered over and over.

We do see, even with those intermediate users going in, even though the content itself is something that they can approach, they really are able to take off and be much more efficient with their time, especially if they're coming in and out, where they might have to set aside R for a bit to do other business activities, and they come back where those routine memorized motions easily fall away, but as you establish those foundational mental models, they stick for a lot longer.

If you are that intermediate user thinking, is Academy, how would I use this? There is something to be said for those intermediate or those power users where you bridge those fundamentals in, and you can even use that to help train people within your organization as you yourself sharpen your mental models, it makes you a more effective mentor and enabler throughout your organization.

Thank you. I'm noticing there are a lot of anonymous questions over on Slido, and one of them was, do you do, or can you do a Shiny course? Thank you for whoever asked that, I hope it wasn't my boss. We are developing a Shiny course, it's still in production, but it's not ready yet. It will be soon, maybe by the end of the year.

Sounds great, and then we touched upon this a little bit in talking about different customers we worked with, but how specific can projects get? The projects are all very similar in that there's a dataset, you're going to visualize it, you're going to wrangle it, you're going to report on it, you're going to fit a model to it. What's in the dataset can be anything, it's up to you.

As long as it has several variables, and it's nice if one's continuous and maybe one's discrete and maybe there's a binary variable, but I think you have a sense of what a rich dataset would look like, and then if the modeling method that you want to apply to it, maybe this is really what matters to whoever asked the question, if it's something that you would cover in a Stats 101 course or even a 201 course, we can go ahead and teach that right now.

We will eventually be developing a course based on tidy models and machine learning where we can go much deeper, but for the course we offer today, we don't teach the statistics or the machine learning, unless it's something, some low-hanging fruit like a Stats 101 or 201 course. So time series, sure, survival analysis, yeah, linear models, ANOVA, logistic regression, that's all the level we're teaching right now.

Thank you. Another question I was just copying over from Slido. Somebody had asked, how does this compare to platforms like Datacamp?

Favorably. We're really focused on that cohort interaction, that human interaction, creating essentially a de facto community of practice amongst the students with their mentor, empowering that to continue within the company afterwards. The motivation and the engagement, from what I've seen, is night and day different. People are really motivated by social pressures and also the identity that they build by engaging in a group that's learning data science.

So I think Datacamp is, and platforms like it, are really great tools for people who are enthusiastic and motivated, have time to spare, self-learners. And I know people who are like that. I'm like that too. But I'm not like that when things get busy or hectic or whatnot. And what I see is engagement does drop off if you don't have that group of people who are pushing you through. And that's what we're trying to provide here.

So I do think that you will notice a difference, but I don't really feel that comfortable talking about check comparisons.

Thank you. I'm seeing a few questions into the YouTube chat right now as well. I think a few of them are around, like, is this only for organizations or can individuals use Datacamp? I mean, sorry, instead of using Datacamp, can you use Academy as well? Right now it's for organizations. We are expanding it to individuals, not just yet, but look this year for it to expand to individuals or groups of two or three even.

Well, we're throwing me off with some of these anonymous questions here. All right, let me let me jump back to some of those anonymous ones and sort through.

Let's see. And then just to confirm, somebody had asked, I assume because you showed the Dow example that this can be industries outside of life science, but I just wanted to make sure that we answer that question here for the person who asked. Absolutely, yes. Anyone who wants to use R, we can teach you.

Thank you all for these questions here. I'm sorting through from YouTube, from some of the comments, too. And then another question was just like around the scope of what we can teach with Academy. What about a topic like package building?

That's great. So Academy is young, as you might be sensing, and we started with how to do data science with R, tidyverse, R Markdown Corridor, that sort of thing. We have a roadmap that we're fleshing out. The next course is going to be programming with R. And I should say we're doing everything, we're doing Python in parallel. So everything I say here about R also applies for Python. So that's the course we're in right now, how to work with functions and program with R, which also will get into package building.

That is a prerequisite, I feel, for actually learning Shiny, for learning tidy models, for learning points more advanced, and that's where we'll be going next. So today, I couldn't sign you up and start teaching package building this afternoon, but if you are interested in learning that, maybe in Q2 or Q3, it'd be totally appropriate to come talk to us, and we'll tell you what we could do.

Sounds great. Thank you. There's one other question that I want to make sure I got to as well, and I'm not sure if I can pull it up on the screen right this second, but Leonardo had asked, is Posit Academy also interested in organizing a sizable amount of courses for people who do not start from scratch?

We are, but the market for people starting from scratch is much larger. There is a much larger need for people, for example, who've been using a different language like Sass their whole time to do stuff, and now they need to do it all in Argo class. We're serving that, but that roadmap that I laid out, you see it will get to machine learning, it will get to Shiny, it will get to more advanced topics.

Every time I talk to someone, I wish I could just jump to what they wanted to learn today, but we're trying to do the most good in most chunks as we go, so.

Okay, I said one more question before the top of the hour, but I think this is an important one to address, and I see Devin, you just answered in the chat too, but Ramil asked, are there elements of building data pipelines and best practices, foundations for clinical trials? Devin, do you want to answer that one?

There's kind of like two elements to it, like to me, when I think of the conversations around data pipelines, we think of like the orchestration components of like, I have a variety of things, I have some raw data, it gets processed a bit, you have some derivative data that you're going to then tabulate and visualize and do some other modeling activities potentially against, you know, Academy won't necessarily cover like how to deploy that data pipeline with whatever orchestration system you might use, whether that be like RStudio Connect or Posit Connect or some other pipeline, but inside the pipeline, the analysis activities, the managing and munging of the data, the visualization, the tabulation, those are the core concepts that are covered with that, so you should be set up to be able to think from like, I have raw data, I have insight objectives on the other end, like, what are the pieces that need to stack together to do that? Those are well within Academy's blue house.

I have raw data, I have insight objectives on the other end, like, what are the pieces that need to stack together to do that? Those are well within Academy's blue house.

Thank you very much. I'm going to do a double check here. If you have a question that you really want to make sure it's answered, if there's anything that I missed, if you want to put it into the YouTube chat here again.

I did also mention in the beginning, when I was introducing the presentation that we've recently had a number of great meetups and data science hangouts featuring different pharmaceutical leaders, and a lot of the questions from those events have been targeted to learning and how to teach groups of new users who might be moving from SAS to R. My colleague Isabella recently put together a great blog post that kind of highlights each of those different leaders' perspectives on moving to open source in the pharma space, and so I just want to put that in the chat there, because I thought it might be really helpful for people listening in today.

One question, let's see. I think if there were some, I know we're a little bit over, if there are any other questions or anything that you missed, I'm going to take some time, that I missed, I'm going to take some time to go through all those questions, and I will get back to you. If there's anything that's not answered, and you want to reach out to me directly, I'm also going to just LinkedIn in the chat here for you all as well. Always feel free to connect with me. I love to meet you all, and always happy to connect you with your teams here at Posit as well. I'm sure many of your companies are working with some of our customer success reps too, so thank you all so much for joining, Devin and Garrett. Any last final words to share?

Well, I appreciate the messages of support that have been shown up in the chat. I've seen all those, and thank you very much, everyone. Thank you all so much. Have a great rest of the day.