image: thumbnail.jpg
Ben Arancibia @ GSK | Data Science Hangout
Ben is a data scientist with 12+ years of technology driven solutions experience in the public, private, and social sector. Currently, a Director of Data Science at GSK focused on data science enablement and open source solutions. Considerable experience in implementing data science solutions, either in the form of transformational efforts in data science strategy, new market entry or new business model development; to more tactical changes, such as machine learning optimization, data visualization improvements, technology platform development, system integration, and operational risk analysis.
_______________________________
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Well, hi everybody, welcome to the Data Science Hangout. So nice to see everybody here this Thursday, feels like this week's flying by. If we haven't gotten a chance to meet before, I'm Rachel. I am clearly the host of our Data Science Hangout, but I also lead customer marketing at Posit.
The Hangout is our open space to chat about data science leadership, questions you're facing and getting to hear what's going on in the world of data across different industries. So we're here every Thursday at the same time, same place. So if you end up watching this on YouTube at a later date, and you want to join us live, there'll be a link in the details below where you can add it to your calendar too.
But at the Data Science Hangout, we are all dedicated to making this a welcoming environment for everyone. We love hearing from everybody, no matter your years of experience, your titles, industry, or even the languages that you work in. So it is totally okay to just listen in here if you want, maybe you're on your lunch break or taking your dog for a walk or something. But there's also three ways you can jump in and ask questions or provide your own perspective too. So you can jump in by raising your hand on Zoom. And I'll keep an eye out for raised hands. You could put questions into the Zoom chat. And if it's something you wanted me to read out loud instead, maybe if you're in a coffee shop or something, just put a little star next to it in the Zoom chat. And then lastly, we have a Slido link where you can ask questions anonymously too. And I think Curtis will share that in the chat here if he hasn't already.
But something I forget to say sometimes, but if you are on a team that's hiring, feel free to share open positions or roles in the chat too. I love to see all the conversation that happens in the chat during the Hangouts, even if I can't read it while it's happening.
But with that, I am so excited to introduce my co-host for today, Ben Arancidia, Director of Data Science at GSK. And Ben recently spoke at POSIT Conference on the need for speed and accelerating R and other open source tools for delivering clinical trial submission. So I'm excited we may get to cover a few of the unanswered questions from Slido during this Hangout too. But Ben, I'll turn it over to you to introduce yourself and share a little bit about your role and maybe something you like to do outside of work, probably also come up with the pronunciation of your last name. I'm sorry I didn't do that with you ahead of time.
No, it was a good pronunciation. It's exactly how it's spelled. So my name is Ben Arancidia, I'm a Director of Data Science at GSK. I sit within our biostatistics organization, specifically within a group called SDSIH, Statistical Data Sciences and Innovation Hub. My role is really focused on what I like to call R enablement. So I spend a lot of time working with different business groups to understand how they can use R for the creation of outputs to regulatory agencies or talk to them, understand sort of what are their needs in terms of tools and then help to build tools for those specific business groups. And then just kind of trying to understand like what are problems that people encounter. So I spend a lot of time actively listening and bringing that feedback to different groups to help kind of solve those problems.
In terms of what I like to do outside of work, I'm a big runner, specifically trail running. I do trail like ultra marathons through mountains, things like that. So that's what I really like doing outside of work. I have two young kids, so pretty much I'm either at home with them or outside running, you know, on some of the trails around my house. So, yeah, that's a little bit about me.
That's awesome. Well, thanks for joining us, Ben. And ultra marathon, how many miles is that?
So the race I just signed up for is in March of next year. It is 55 kilometers, so 30, 33, 34 miles in the mountains near Asheville, North Carolina. So yeah, it's called the Mount Mitchell Heartbreaker. So I hope it doesn't break my heart, but we'll see how it goes.
Impressive. You'll have to let us know how it goes.
From consulting to pharma
Well, Ben, something I thought might be interesting to kind of kick off the conversation with as we're getting questions from everybody here is I know that you were a consultant in a previous role. It might be good to hear a little bit about your journey, but I'd also love to hear, like, what was it like transitioning from data science as a consultant to data science in the pharma industry too?
Yeah, so how I became a data scientist is pretty funny. I went to a university, I studied, like, quantitative economics and geospatial information systems and became a GIS developer right out of college working for the United States Postal Service on a federal contract. And I worked on, like, route optimization. And I was like, this is pretty cool. I really like doing this. And then the term, like, big data came around and people became data scientists. And I thought to myself, oh, wow, that's pretty cool. Maybe I should just do that instead of, like, being a GIS developer and kind of really get into data. So that's what I end up doing is deciding, all right, I'm going to focus more on data. I can always go back to GIS later, but really focus on trying to understand, like, how do I deal with really large data sets to solve different problems?
So I ended up becoming a consultant. I worked in commercial consulting specifically for cybersecurity clients, building out, like, different tools, platforms, implementing machine learning methodologies, things like that. And it was a really great way to get exposure to lots of different problems as well as being able to build up my skills for things, like, I might not have learned at school.
And eventually what I ended up doing is transitioning into the pharma industry through a great leadership program that GSK has called the ESPRI program, which basically looks to bring outside talent who has a specific knowledge base and bring it into GSK that they want to expand upon. So my cohort of ESPRI candidates, we're all data scientists, and GSK Biostats wanted to bring data science expertise in-house, and they do it through this kind of three-year leadership program. So what we do basically is we do rotational programs through different groups within the Biostats organization, bring our data science expertise, and then they teach us the pharmaceutical industry business essentially during those three years. And then at the end of the three years, you find a final position within Biostats.
But it's a great way to think about, oh, this person has some, like, interesting talent that we want, but they don't have, like, the business knowledge. How do we kind of marry the lack of business knowledge with the talent that we want to build? And I think that's one thing that I really love about GSK, and sorry to be like a GSK promoter right now, but being able to have that kind of space to be able to learn and develop is really crucial to being able to create, you know, those good outcomes for the business.
Yeah. I'll echo what Ben said. It sounds like a great program that a lot of other companies may benefit from, too. And also realizing there's a ton of love for geospatial in the chat here, too.
That's cool to see about all the people that love GIS.
Rotations, training, and open source in regulated industries
I'd love to hear a little bit more about, like, what it's like going through those rotations, too, and teaching people data science. I imagine you're doing a lot of training. And what it's like, like, them teaching you the pharma side of things as well. Could you give us an example?
Yeah. So one of the things that came up a lot of different times was really, so if you think about pharma, we're a highly regulated industry. And I think one of the things that one of the conversations I often had with people was, well, we can't do this because we're a highly regulated industry. And to go back to your previous question about, like, what I learned from consulting, because of the nature of my work previously, I worked in highly regulated industries, but we were able to use open source tools. So I think one of the things that was really interesting going through the rotational program and talking to different business groups and things like that was being able to have sort of that outside pharma industry perspective to say, like, look, you're highly regulated, but so is, you know, a lot of automobile companies or oil and gas or, you know, X, Y, Z, other companies as well. And they're able to use open source tools. I think, like, you might have to think about it differently for your specific industry.
But, like, you know, if you look at, like, pure accounts of regulations, I think automotive and oil and gas and financial services all have more laws and regulations compared to pharma. So I think one of the things that was interesting was going through those rotations and consistently having that conversation that, like, you don't just default because you're highly regulated. You can't use these tools. Just figuring out how do you use them safely within the organization to make sure you're satisfying those requirements.
So I think one of the things that was interesting was going through those rotations and consistently having that conversation that, like, you don't just default because you're highly regulated. You can't use these tools. Just figuring out how do you use them safely within the organization to make sure you're satisfying those requirements.
I got excited by you describing that program, and I jumped way ahead to what you learned as a consultant. But are there other things you wanted to expand on about that?
What else I learned as a consultant? How to tell a good story. I think that's one thing that constantly when I'm working with data scientists who might be newer is, like, how do you tell your story about what you learned during your analysis or the outcomes or the outputs from something? I'm sure it's a trope by now, but, like, communication and being able to tell a really nice story about what it is that you did, what is it that, you know, would improve a decision maker's, you know, life is crucial. And I think one of the things that data scientists or any sort of analytics professional has to balance is, like, not only can you do the technical work, but can you do the communication and can you tell that nice story arc?
What's one of those communication or storytelling tips you'd share with us?
Less is more. I think, like, one of the ā I think if you can have, like, to go back to PositConf, for everyone's awareness, like, if you do a talk at PositConf, you'll go through, like, speaker training. And one of the first trainings or maybe the second one or maybe it is just the first two, you focus on figuring out, like, what is your governing idea? Like, what is the one idea that you want everyone to walk out from after seeing your talk? And I think that should be applied to how you communicate with stakeholders. Less is more. Like, figure out, like, what it is that you want someone to walk away with based out of your meeting. Because if you're trying to show, like, a presentation, that presentation is not the only thing that that stakeholder is going to do in that day. Like, they're going to be receiving, you know, hundreds of emails per day. They are going to have to be making decisions, things like that. So the question is, like, how do you make sure that they get that one piece of crucial information you want them to have and really focus on that? So less is more. What is your one governing idea? And then everything else is just kind of, like, you know, extra gravy if you can get it. But figure out what is that one thing in order to make sure that people walk away with it.
Absolutely. I know a lot of people have joined since the, like, first beginning when I say how people can jump in, too. So I like to just sometimes recommend or remind people, if you have questions or things you want to add to specific topics, feel free to jump in by raising your hand here on Zoom. You can put questions in the Zoom chat. And if it's something you wanted me to read out loud instead, you can just put a little star next to it. And then Curtis or Tyler will reshare the Slido link in a second, too, if you wanted to ask anything anonymously as well.
I realize sometimes I say that too early when not everybody has joined. But Yanan, I see you asked a question here in the chat. Do you want to jump in?
Hi. Hi, Rachel. Hi, Ben. Hey. Nice to see you. Yeah. I have a question regarding the R validation for the regulatory submission to FDA. In Medtronic, we usually need to do R validation after we have updated version of R. I don't know if JST has this requirement to do R validation for the submission work.
Yeah. So, GSK, we do R validation. A lot of how we do R validation can actually be found on the R Validation Hub, which I think is part of the ā is either a POSIT working group or is loosely related. So, how we do it is really kind of taken from there in terms of how we do it, the R Validation Hub. I can try to find the link off the ā
Oh, I'll share it for you right now. Okay. Perfect. Yeah. So, that's ā Connor beat me to it. Thanks, Connor. There you go. So, yeah. So, we generally do that. We were a ā we, being GSK, were a strong participant in figuring out how to do it. And then we worked with basically our QA department to make sure that we're checking all the boxes that we need to hit in order to say that, yes, we believe that this is validated.
So, for those who are not in the pharma industry, maybe I'll explain what validation means because it's kind of like a ā it's an interesting problem. So, within a biostats organization, you basically have two things that you need to accomplish to move assets or drugs through the pipeline. Basically, one, can you tell what ā can you trust the efficacy of a drug and can you trust the safety of a drug or an asset? And how you do that is you write statistical code in order to be able to prove that, yes, we trust these statistical tests that we've written.
In order to do that, you need to do something that's called validation for your software, which is essentially to say we have tested the software and we believe that all the outputs from this software is correct. How you do validation is defined by the organization, but it's really important for you to be able to trust your software. So, there's a lot of testing that goes into place in order to get that level of confidence within an organization that says, yes, this software can be trusted. And there's different nuances for how you do the testing.
So, for example, if you're using ggplot2 for creating outputs of graphics, things like that, you can generally test it through code coverage as well as basically doing a couple simple simulations and saying, okay, does it actually create the output that we would expect if you were to put it ā basically create that output? So, there's that. But then there's also output ā there's validation from a statistical correctness point of view. So, for example, if we're writing a piece ā a package to do linear regression or doing a linear regression ā or using a package for linear regression, how do you test for that statistical correctness? And that's a little bit more of a thornier problem to figure out, but it tends to require like a lot of different simulations, things like that, in order to be able to say, yes, from a statistical correctness point of view, we trust this piece of software.
And that's a little bit more of a thornier problem to figure out, but it tends to require like a lot of different simulations, things like that, in order to be able to say, yes, from a statistical correctness point of view, we trust this piece of software.
Hosting and environment management
Thank you. Thank you, Obed. I appreciate the explanation there, too. I see Travis has a follow-up question around validation. Yeah, thanks. Hopefully, it's not too windy out here. I had to take your walking the dog thing literally. I probably needed to go out. So, anyway.
I was also going to say, does anybody remember when we were talking about all getting hex tattoos at a previous data science Hangout? Travis actually did that at the conference. Yeah, it was fun.
Hey, Ben. So, when you validate ā I mean, where are you all hosting your R? Is it like Workbench? Do you run it in the cloud? Are these like Docker containers, local machines? Like, what do you do for that? So, what we do is we have it ā we host it on Workbench, and what we do basically is we create what's known as a frozen R environment, which is basically an environment that has a set number of packages that we have gone through the validation process with and said, yes, we trust these packages.
So, then what we do is then we create basically a frozen R ā I don't know. It's not truly a container, but basically an execution script where we can then basically just take and install that in any system that we choose. So, the goal is to make it system agnostic. So, for example, if we want to install it in like in Posit Workbench on our own internal, you know, our own internal RStudio cloud environment, we could do that. If we want to do it on, I don't know, pick your data science Workbench, Domino, I don't know, whatever, different things like that, you're able to do it, but trying to make it system agnostic.
Yeah, I mean, there's like two levels to the validation. It always seems like validating the packages inside of R and then validating the installation of R itself. And that latter one is the one that, you know, we've had the most challenges with, you know, how do you prove that R is actually R, is sort of the philosophy, the spirit of what FDA wants to see, and we find that surprisingly difficult. I think they're coming around to it, though. If you see some of the talks that they've given in terms of like the R adoption series and things like that, I think they're, and also, I mean, frankly, if you just look at like statistical master's programs and things like that, like R is now kind of the tool to use. So I think from like a, call it a culture point of view, I think that's changing pretty, pretty rapidly just because of just how the workforce is changing.
Hey, Tim, I see your hand raised. Yeah, just talking about the whole package and R validation front. I think that's one area where the combination of Workbench and Connect I've found really helps because when I'm publishing to Connect using PackRat, it's making sure that everything is valid, that it's a snapshot of the packages that were used to publish it, and even though I upgrade R on the host machine and deploy other packages or other apps with different levels of software, that one that I encapsulated when I published it before stays that way unless I make a specific decision to update components. Just a plug for integration of Workbench with Connect.
Thanks, Tim. I see Priyanka had a kind of a follow-up question on this topic too. Do you want to jump in here, Priyanka? I can ask it if you have a spotty connection, but the question was kind of like, well, is that using RM? And then I see Travis asks, PackRat versus RM, I don't know the distinct distinctions of that. Could anyone speak to that?
Yeah, Tim, you mentioned you were using PackRat, and that kind of, we use RM, I think, for the same, but I don't know the difference. I think both are used when it gets published. PackRat manages the package versions and does the uploading, so basically, it's kind of like a docker light for your packages that makes sure that all of the revs are there, and it keeps a library in the background and makes sure that everything's synced up. And I would imagine RM is used as well, but I don't know the technical details of that. One nice thing about publishing Connect2Connect is I don't have to, it just does it on the background. But if I was writing it myself, yeah, I'm sure you'd have to use a combination of both if you weren't publishing Direct2Connect.
Pharma principles and reproducibility
I can share a link in the chat in just a minute about this too. But Lisa, I see you had a question a bit earlier about being outside of PhRMA and how things might relate. Do you want to ask that? Yeah, I mean, I think I'm both curious about just like sort of learning the PhRMA lingo, which has, like every industry, a lot of abbreviations to it. And then just really also, you know, sort of trying to understand that whole process and think about how those same ideas might, you know, even though maybe I'm in an industry that's less regulated, how some of those ideas might want to steal some of those because it might just be good practice in general. So just kind of where to, good places to start learning.
And I already saw the, our PhRMA conference posted out there. So any other ideas would be great. I think at the core of the PhRMA world is basically, can you reproduce your outputs and can you trust and verify those outputs? So like you could learn, you could learn like the PhRMA lingo if you want it. But I think the core thing is like, do you, can you figure out how to create reproducible data pipelines essentially? And I think if you are able to do that through different methods, as well as using sort of like potentially like RMF in terms of like how to manage your environment and things like that, I think you actually will, you actually know more, like that's essentially the core of what is PhRMA principles essentially is how do you reproduce, how do you make sure that it's reproducible and how do you make sure that you can trust and verify your outputs? And I think that's kind of the core.
I think at the core of the PhRMA world is basically, can you reproduce your outputs and can you trust and verify those outputs?
So I have to learn REND. There are definitely a lot of great resources on RMF. Maybe someone in the back around wouldn't mind sharing some of their favorites in the chat too. That'd be great.
But I did want to ask, I put this in the chat, but also because it just came out, just came up, give a shout out to some of the free workshops that are happening this week for the RNPharma conference, which is open to everybody, regardless of if you are working specifically in that field. But there's a ton of great workshops all this week, and then the conference is next week. And so I shared that in the chat just a second ago.
Ben's role and team structure
There was another question a bit earlier, and I'm not finding it specific exactly, but the question was, Ben, can you give us a little bit more background around your role too and what your team looks like? Yeah, sure. So my role is really, I like to split it up into, call it four pillars of responsibility. I have a team that focuses on building sort of end-to-end applications for trying to gather insights about data related to either clinical trials or some of our business partners outside of biostatistics.
One I can give a shout out to is called the BPH tool, which is an externally facing tool on how to treat a certain urological, it's not disease, but called symptoms. So I'll put that in the chat. But we built that using Shiny. It's linked to like scientific manuscripts. And so one of the things that we try to do now is when we submit a manuscript, try to either submit a interactive tool or enhanced graphics with the manuscript in order to make it easier for people to understand the data. So that's one team.
I have a team that focuses on building like open source packages. One of the big things we're looking at now is like Bayesian statistics. And I have a third team that's focused on doing basically R training as well as R enablement. So we have a program called Accelerate R, which is a group of agile, it's an agile pod of data scientists and R programmers that go basically study to study, sit with them for nine to 10 weeks, train them on the use of R, and then basically leave after those 10 weeks. But really set up those studies to be able to use R effectively.
And then I have a fourth kind of like, kind of pillar of responsibility, which is focused on being engaged in like data science working groups, and different things like that in order to be able to, to talk about R as well as be able to have a kind of industry standard for how to use R for different ways for clinical, for clinical trials. And we talk about things that, you know, might sound silly, like, you know, how do you round in R versus SAS and Python, but being able to talk very confidently about those differences and why you might get differences in results is a pretty crucial aspect for our industry.
Scaling R training
I know you just mentioned the training piece, but I realized that's like a huge undertaking with so many different trainees. And I was wondering, how do you scale that training out? I think when we talked, you said that there's like 800 trainees? Yeah, so within Biostats, GSK Biostats, we're about 900 people. So what we started to do initially was having just open in-person as well as like virtual trainings with R. So like, I think we got to the point where 80% of Biostats took an introduction to tidyverse R training, which is like, you know, 80% of 900 is like right around 700, 720, something like that.
And what we were finding was people just weren't uptaking the training and using it on clinical trials. And the big reason we found out was because you would take the training and then you wouldn't actually use the, you wouldn't use R until like 12 or 18 months later. So what we ended up doing was deciding like, all right, let's just put R training as on demand. Like, I'm never going to write a better like intro to tidyverse training than, you know, either Hadley's written or someone else has written online. Let's just make it all on demand when you want to do it and focus purely on like support and mentoring.
So what we do is we have this like small group of people and they go, you know, basically study to study whoever wants to use R at that period of time and we work with them. And the goal is basically train that study team up and create them into like champions and then also give them the confidence to be able to go to other studies and help provide guidance as well. So we do it through that kind of support and mentoring, but we also have like big community engagement through different like open forums where people can ask questions, things like that. So it's really focused on like spending less time on creating materials for training and documentation and more time on figuring out like, all right, how do we actually do support and mentoring as well as get feedback to build tools that actually will help our users for actually using R on clinical trials.
And Ben, I see you have a question in the chat. You want to jump in here? Yeah, I was curious, Ben, if you have other cross disciplines across GSK as it's so large. I mean, obviously you have a large enough group you're working with to try and ramp them up, but it seems like there could be other opportunities that would enhance the efficiency of the tools and so on and so forth.
Yeah, there's lots of different data science groups. So like I feel like, you know, sometimes you just like stumble upon them because I think GSK, I think we're like 100,000 people or something like that, maybe a little bit less. And so you're just like, you're just like, you just stumble upon them.
For Biostats, we're pretty large, luckily. So we work a lot with our business partners. And the reason why we work with them is like, because Biostats really focuses on moving assets through the pipeline, we have like direct impacts on like HR or finance. So like we have those connections that we do, but like we have a whole like AI group that's focused on like computational biology and computational chemistry and different things like that. So there's like lots of different data science groups within GSK.
Some of them work across departments, but a lot of them tend to work like within their own department because they're so hyper focused on like specific problems that they're trying to solve. I was talking to someone at the conference who works in pharma and hearing about some of like the different finance use cases within pharma, which I wouldn't initially think about as well. Yeah. It's cool to hear about all the different data science teams across companies.
Redefining vs. adding tools
This was actually a Slido question from your conference talk, which I thought, because we didn't get to it, we could cover it here. But it was, how do you think about redefining the way that people see themselves, like helping statisticians see themselves as programmers and using Git, for example?
Yeah. So I actually don't believe in having people redefine themselves. I actually think about how do you help someone like learn a tool or add a tool to their toolkit in order to like enhance the way that they're working? Because at the end of the day, you are not going to turn like a statistician into a programmer or a computer scientist or things like that unless they really want to become one.
So I think the goal is like you don't actually try to redefine them. I think you try to help them add tools to their toolkit in order to be able to use it in order to enhance their work. So like teaching like a statistician Git, like you have to explain why it's important for them to know, you know, kind of the hit by the bus rule, if anyone's familiar with that.
I think you try to help them add tools to their toolkit in order to be able to use it in order to enhance their work.
But yeah. It's time to use the what if you won the lottery. Or yeah, what if you were hit by a bus. Yeah, that's a better way to put it, I guess. But yeah, so like I think it's more focused on like, how do you help people add and not necessarily redefine? Absolutely.
Rolling out the training initiative
And is this the training initiative, something that's kind of come from the top at GSK? Or how was it first rolled out? Yeah, so I mean, it was more it wasn't necessary. It wasn't really rolled out from the top. It was more like, we got the blessing from the leadership team that says, hey, we want to try something pretty different. Are you okay with this approach? And got the thumbs up.
And so we got the thumbs up to say, yeah, go and go and try this new this new way to train focus less on this on the classes focus more on support and mentoring. And that's how we were able to do it. So it was luckily we were given the freedom and the latitude to be able to try something new. And, you know, in the spirit of agile, fail fast if it didn't work, but it has worked pretty well. So luckily, it wasn't something that had to be top down driven. And I think it's a model that could be utilized pretty much in any organization.
Because really, all you need is like people who are passionate about are and willing to spend some time to do that, that support and training. So yeah.
Rounding differences between SAS and R
I see like a pretty big conversation happening in the chat around rounding. And can somebody explain to me, I've heard of this and like how numbers can be different and SAS versus R, but I would love to understand a little bit more of what that means. I see a bunch of blanks being here.
In general, someone someone correct me if I mess this up. But in general, computer science has adopted the strategy that if you have something that ends in 0.5, like 2.5, you round the even numbers down. So like 2.5 becomes two and the odd ones round up. So 3.5 becomes four. And so what that does is on average, if you have like a whole bunch of numbers that end in 0.5, then the rounding up and down like on average doesn't really matter. SAS, on the other hand, is the only language that will round everything in 0.5 up. So if you had a whole bunch of numbers and you rounded them all, you would actually get an average that is higher than the true average was because everything got rounded up instead of like every other one getting rounded up.
And it's a very annoying nuance for submissions to like FDA or sponsors or investigators when they're used to seeing SAS outputs and everything that's 0.5 being rounded up. They start to question that. And indeed, at the FDA, if you submit something in R, you might get a mismatch in SAS because of that rounding problem. It's a weird one. Do you have to write like custom functions to do R rounding like SAS in order to get around that or just kind of explain it away via differences? Yeah, those do exist. And Stephanie in here shared a really good resource that looks like some packages and other people developed around this idea. But yeah, for sure. There are custom functions that we sometimes write. If they really just insist that everything 0.5 rounds up, then we go, yeah, OK. And we just note it. Thank you for that explanation.
Yeah, sometimes it's like if something's been said like enough times, it's like, do I pretend that I know what that actually means? Or do we just stop and ask?
Measuring success of the enablement program
Catherine, I see you had a question here in the chat. Do you want to jump in? Yeah, you've mentioned that you've switched kind of the way that you're training from a previous method and that it's been successful. But how are you measuring that success? We're currently rolling out like a data science mentorship program and trying to figure out how do we measure how successful we've been at mentoring this person when we don't necessarily want to turn them into a machine learning engineer or a programmer or something. We just want them to be better data scientists. And data professionals.
Yeah, so for us specifically, we're pretty it's it's we try to keep it very simple, which is basically like how many outputs are we creating with R and is that number increasing? And if the answer is yes, then we're hitting our metrics essentially. Or is the number of clinical trials that we want to use R in some part of increasing? Then the answer is yes. We believe that we are we are hitting our our our tracking or metrics in order to say yes, our enablement is is increasing.
If you're talking about like, how do you make someone? How do you make some? How do you track if someone is improving their skills? To me, that's more like a development plan kind of point of view at an individual level. But I think it's really hard. I think you can only do it in terms of in terms if you really want to track, I think it has to be sort of like at an aggregate output level and not necessarily did this person like comprehend unless it's like at a development or a career or personal development level. If that makes sense. Yeah, I think that makes a lot of sense, especially given that your team's goal is R enablement. That's something that's fairly straightforward to track.
Are we seeing a higher use of R? Yeah, we're doing the professional development part was was a lot harder. We had, you know, a couple ideas, but really, you know, one of those things is are they using it now? And like, that's something that we're that we're tracking as part of that, too.
Yeah, that gets really hard having gone through like those wars with like trying to teach people how to use it like it's it's the steepest learning curve that I think I've ever encountered with with with teams. It just it's really, really tough. And there's just like so many points of friction, just like when you get your first merge conflict and you just don't know what to do. And it seems like the world just like will not let you merge your, you know, your branches together like it is a really, really tough thing. So you just unprotect your branches and merge them anyway. It's fine. Yeah, but then you got to explain it to someone later on reproducibility, right?
So I see Rebecca had a question. There's a star, so I'll read it, but it kind of expands on this. But how are you demonstrating or showing that evidence for it working? I guess to expand on this, are you presenting this to anybody as well? What specifically in terms of I think about to expand on like how you're measuring or tracking the success against the previous method?
Yeah, I mean, I don't think we I mean, we're not trying to get to a world where R replaces SAS. We want to live in a world where you can select the tool that you want to use to create your outputs. So if our goal is to make R a primary programming language or a tool that can be used, we should be and there's an interest and we should be able to see increase of the use of R. So like I don't look at it as like as a I look at it as like you have options and we want to make sure those options are being utilized and not like how do I get R usage from, you know, 10 percent to 100 percent within my organization.
Cultural barriers to open source in pharma
Thank you, Kevin. I see you had a question around the cultural barriers. Want to jump in? Yeah, sure. Yeah, I think you spoke a little bit to this, but but I was just curious if you could elaborate more on how you kind of dealt with cultural barriers to adopting open source, you know, in pharma, especially like, yeah, it's a regulated industry. And and I almost want to frame it like like was it more helpful to show like the efficacy of open source? Or is it more important to show the safety of it? You know, like or there's both kind of important, you know what I mean?
Yeah, yeah. So the the big thing that actually was the major major way that we broke a lot of ground in terms of open source was visualizations. So because one of the big things that R does so well is is ggplot and being able to create like awesome visualizations, you know, with data. And that was something that like SAS really, really struggled with. And so the wedge that we actually figured out in order to be able to start having the open source conversation was like, look at all these cool like visualizations and look how easy it is to make these visualizations compared to like other other programming languages.
Why don't we like start looking at open source as like a way to be able to to you to basically use tools successfully? And then a lot of things happen. But I think the big another thing that really has broken through cultural barriers has been there's a really, really strong industry call it collaboration called like the farm reverse where different and different companies. So like Roche, GSK, Novartis, J&J, like all these different companies are creating open source packages for the clinical trial development pipeline. So like if you can set if you as an industry can agree upon, like we all trust these open source tools.
It really kind of helps to break down those cultural barriers about like, why should I use this open source tool? And it's it's really easy to say, like, well, we want to use it because, you know, Roche is using it or we've developed it with Roche and with, you know, insert other pharma company. And you can say we we we helped create it. We should trust it. And so I think that has been one of the things that has helped with those cultural barriers.
And it's it's really easy to say, like, well, we want to use it because, you know, Roche is using it or we've developed it with Roche and with, you know, insert other pharma company. And you can say we we we helped create it. We should trust it.
Just one follow up. Did you ever hit a wall where like there's some parts of the organization or some people, you know, some managers who just like can't get over the, you know, that that the validation or the kind of concerns about about open source? Or did you find you were able to pretty much break through it when you needed to? I think you can break through, but at the end of the day, like I tried to talk a little bit about earlier, like if people want to use their proprietary tool, fine, if it helps them do their work. Like at the end of the day, like all I care about is like, can you do your work and can you do your work in a timely manner and not have like bottlenecks with platforms? And if the answer is yes, then like, who cares? You know, if that's your preferred tool, that's where you like to spend your time, then like I'm not going to tell you differently. That makes sense. Thank you.
Thank you all for the great questions as we. If anybody feels like their questions are getting missed, I'm sorry if we missed them in the chat, so feel free to like repost below in the zoom chat as well.
What the FDA wants to see
But there was one anonymous question that was. When we sit, when we say like what the FDA wants to see, like, what does that actually mean? Like, did those statements come directly from FDA employees or where do people hear that in terms of, I mean, you could talk a little bit about the FDA in terms of like, I think there's like guidance on software. Like if I was Phil Boucher, I would like have the PDF up already on like what it is that you can use.
But I think there's a lot of like when you submit something to an FDA or another regulatory agency, it's like an open conversation and you have a conversation. And so it's not just like it's not a piece of homework where I have to submit this at midnight and then I get a grade. There's like an ongoing back and forth conversation that occurs. And so being able to being able to engage and have those conversations is where you learn exactly like what it is that, you know, is needed from a regulator. Now there's like expected standards and you know what standards it is that you have to
hit in terms of like having a successful submission or or ad hoc analysis or whatever it might be to be able to answer, you know, a scientific question.
But it's not like I said, it's not a it's an ongoing conversation. It's not like a submit your homework and you and you hope you get a good grade.
Ahmed, I see you had a question you just put into the chat. Do you want to jump in here?
Yeah, sure. For the different Bayesian statistics that could be used, I gather it's going to be different between discovery and drug discovery or clinical trial data analysis versus sales or things of that nature. And I'm not sure if Ben deals with both.
But I was curious if if some of the algorithms that are proprietary could or have been used in R or developed in R.
I can't speak specifically about Bayesia Lab. I can tell you our approach to Bayesian statistics is open source. It's published in different journal articles, things like that. We focus on Bayesian dynamic borrowing. That's like our our focus in terms of how it is that we use Bayesian stats. But I can't I don't know enough about some of the stuff that goes on in Bayesia Lab to be able to answer that question.
Travis, I see you put a question into the chat here. I want to jump in.
I'm sorry. I feel like I'm talking too much. Hi, everyone. Again, Ben, we are. Don't worry. I'll stop calling on somebody if they talk too much.
Ben, I'm sure like all of us in the pharma adjacent space, you probably watched the Novo Nordisk talk, the Journey to NR submission. Yeah, you're nodding. Yeah, yeah. It was awesome. We were all waiting for that one. And they're very open for everyone who didn't see it about challenges with submitting things that the FDA maybe hasn't seen before. They're sort of like they submitted several times. It was like, no, this doesn't work on my machine. And then to Ben's point about it being conversation, they kind of had a conversation. They work through it. So I was wondering if you had any like, you know, just gut reactions about that talk, like any further commentary you might call you might want to add to what they showed.
Yeah, there are two things that I really took away from that talk. One, it was a really similar story. We all are solving the same problem, albeit in our different ways, based off like our organizational culture or just organizational preferences.
The big thing I took away from it actually is if we are going down the open source route in terms of R and different R packages, things like that, I think it's actually going to potentially create a stronger culture of open source tools because it seemed like the.
And by that, I mean, I don't know if we're going to if specific companies will be able to have internal tools that only they can use because of the issues that Novo had with Novo had with using their own internal tools and trying to do a submission because of like, how do I get them to be able to install this package, which might not be available on CRAN? How do I be able to do this? So I think it actually one of the things that it's made me think a lot about is like, how much internally do we actually want to create because of like the hiccups that could cause in the actual submission process?
And will that potentially result in like a stronger open source ecosystem because of just like just the making it easier in order to for regulators, whether it be FDA, EMA, Japan, whatever it might be to be able to use these tools?
And will that potentially result in like a stronger open source ecosystem because of just like just the making it easier in order to for regulators, whether it be FDA, EMA, Japan, whatever it might be to be able to use these tools?
That's such a great answer. Yeah, I felt the same. There was that one part in their talk where they where they said, well, the solution was just we needed to open source everything and then the FDA was happy and I was like, oh, I didn't expect that. But it's almost like it's a good advisory tip for those of us developing internal packages at our companies like that may not be a sustainable journey in the future when we go to submit.
Yeah. And I think the big thing that I have been thinking a lot about it is like the reason why we've never encountered this, we being the pharma industry has never encountered this is because like, say, SAS is your tool of choice. Theoretically, SAS is the same from one organization to the other. And I keep harping on this, so I'm sorry, but like reproducibility is key. So if it exists in two or if your proprietary tool exists in two organizations in the same version X, Y, Z, then you kind of have to make sure that that's guaranteed if you're using open source tools. And like from a philosophical point of view, that probably means we have to open source more to make sure that everyone has access and we don't have these like very secret internal tools that cause issues with submissions.
Career advice
Ben, a question I love to get to ask people is about career advice they've received over the course of their career or have given. Do you have something that stands out?
Oh, man. I think the thing that always stuck with me is it's OK to say no. Like you should be ruthlessly prioritizing your time and trying to work on things you're passionate about. So it's OK to say no to certain things if you are the wrong fit. It's also if you think about like your brand management, it's sometimes nice to be wanted. So saying no sometimes is good for that brand management.
So I think being comfortable with the word no and is really important. The other thing and I think is really important or the other piece of advice that was given to me and I try to live by this value is like be radically transparent. So meaning no matter what, just be like just no such thing as like, you know, holding cards close to your chest, basically say like, here's what I got. What do you have? And really focus on being able to have those open and honest conversations, which I think leads to greater vulnerability, which is like how you make connections with people and really are able to be empathetic with people and trying to understand their problems. So I think being able to say no, but also and also being able to be radically transparent is really crucial work advice that people that a lot of mentors have given me in the past.
I try to live by this value is like be radically transparent. So meaning no matter what, just be like just no such thing as like, you know, holding cards close to your chest, basically say like, here's what I got. What do you have?
Absolutely. Thank you so much for sharing that.
Connecting through training and education
OK, I don't want to have I don't want to take the last few questions if there's anything that I've missed here. So please feel free to raise your hand if I have missed one of your questions in the chat, but I'll I'll jump ahead with OK, I'll keep keep an eye out, but I'll jump ahead with one I wanted to ask as well.
I know, like in your role, you get to connect with so many different teams through training and education, and I'm I'm curious how that's helped you in your role and how you build these relationships with people and they maintain those relationships.
Yeah, I really like it because I'm able to understand more about like what do people what are people facing? And one of the things that I've learned is like every. Or one thing, one learning I've had from it is like no one's trying to do a bad job. It's just there's so many things that are going on in life that you have to people require like certain different levels of support. They all want to try. They all want to do the new cool thing. It's just everyone's at different stages.
So I think being able to connect with different teams, understand sort of how it is that they where they are in their point of life or where they are in their development journey in order to be able to do these things. It's really important to be empathetic and learn and be able to take those learnings and bring it to other teams. And I think that's one thing that I find really cool about my job is like being able to connect with these different teams, bring learnings from different places, bring it to other teams, make those connections as well as like bring what are the problems that people are facing to like our different tools teams or to my other my tools team to be able to say like here are things that I'm hearing. What are your thoughts?
And I think like a great example of this was another talk at Posit.com from GSK. We created this open source tool called Slushy, which is built on top of RM to do environment management. So I don't know if we would have ever gotten to that point without being able to go out to these different teams and like actually listen to what are their problems for how for what they're encountering if they want to use R for their for that creation of their outputs.
Yeah, I was trying to go find a link to that talk, but I forgot they're not all public yet, but they will be in the next few months.
Ben, was there anything that like after your talk you thought about? Oh, I wish I also covered this or maybe a question that was asked of you that you had wanted to expand on.
Um, I think the biggest thing is. If you really want to. If you really want to enable learning, you actually need to put the support behind it. And I think the big thing that I wish I was more focused on in that talk was you can do as much training as you want. Um, but if you're not actually using it. In the creation of an output, that's that's where your you actually learn how to do things and there needs to be a really robust support team behind to help answer those questions. And I wish I kind of hit that more is just really focused on that support and mentoring. It's really hard to scale. I realize because it's, you know, actual hours and not, you know, a piece of document that someone can go read, but that that support and mentorship is crucial for for being able to change and enable an organization to use a tool.
And I'm sorry if I missed this in the beginning when you were describing the different roles in the team, but is there like this. Idea of like a help desk or center of excellence or where people go for additional support? Yeah, so that's what that accelerate our team is and we have like a giant community forum where people are able to post questions as well. I think we call it like are for clinical forum and people will just post questions or post announcements about new packages and things like that. So we have, like, not really a sort of the direct support, but we also have, like, sort of the robust community that has like a team's channel where there's an active kind of communication that occurs that answer questions or celebrate wins, things like that.
That's awesome. Thank you so much, Ben, for joining us today and sharing your experience with us. I wanted to there's so many helpful links shared in the chat today. I wanted to remind people if you wanted to save the chat, there's this little three dots in the bottom of Zoom. You can press those and press save chat so you can have all of those links as well.
But Ben, if people wanted to connect with you after is LinkedIn the best place? Yeah, I will drop my LinkedIn in the chat right now. So if you have an interest in connecting and sending me a message, if you have, like, other questions, like, yeah, don't hesitate to connect.
Great. Thank you so much. One thing before people go. I saw Libby had posted this week about something about learning to tidy models, and I saw there was so much interest from the community. And this month, our monthly end-to-end workflow demo is going to feature Simon Couch from the Tidyverse team, who's actually going to be going through tidy models and using that with Posit team. So I just wanted to share the short link to that if people wanted to add it to their calendar. You absolutely don't have to register for anything. It will just be on the Posit YouTube as well. But it seems like there's a lot of interest in tidy models there, so I wanted to share it.
Thank you all so much for joining. Have a great rest of the day. Thanks, Ben. Thank you. Thanks, Ben. Thank you. Thanks, Ben. Bye, everybody. Bye-bye.