Resources
Feb 20, 2024 |
1h 3min
|
517 views

Kevin Snyder & Raju Rayavarapu @ U.S. FDA | Data Science Hangout

We were recently joined by Kevin and Raju @ the FDA to chat about democratizing data science by teaching people the basics of data wrangling, data science and the power of open source and utilizing principles of systems biology, pharmacology, toxicology, bioinformatics, and statistics to collaboratively develop open-source software solutions that support decision making and policy development.

Links mentioned:
πŸ”— FDA Digital Transformation Symposium: https://www.fda.gov/news-events/fda-meetings-conferences-and-workshops/2023-fda-digital-transformation-symposium-12042023
πŸ”— precisionFDA: https://precision.fda.gov/
πŸ”— R4DS Online Learning Community: r4ds.io/join
πŸ”— R Validation Hub Contact: https://www.pharmar.org/contact/
πŸ”— Hangout LinkedIn Group: https://www.linkedin.com/groups/12610075/

Speaker bios:
Kevin Snyder - Associate Director of Nonclinical Informatics at U.S. FDA

Kevin received his Bachelors in biochemistry from the University of Maryland in 2008 and his PhD in neuroscience from the University of Pennsylvania School of Medicine in 2013. He currently serves as the Associate Director of Nonclinical Informatics in the Office of New Drugs in the Center for Drug Evaluation and Research at the US FDA where he manages data science and informatics initiatives to support the pharmacology/toxicology review program. These initiatives include research efforts to develop methods to optimize the regulatory use of standardized electronic CDISC-SEND-formatted toxicology study data as well as internal informatics projects to promote the development of scientifically sound, data-driven regulatory policies. Dr. Snyder also leads an agency-wide Data Science and Software Development working group that is focused on building out the organizational infrastructure necessary to support the work of data scientists across the agency and is an active collaborator with several consortia efforts, e.g. CDISC, PHUSE, and BioCelerate, to improve the implementation and use of the SEND data standard.

Raju (Rama) Rayavarapu - Data Scientist at U.S. FDA

Raju comes to the FDA from the great state of Pennsylvania via South Bend, IN and Memphis, TN where he did his Ph.D. and post-doctoral work. He is the lead of ODAR’s DataForward Initiative which is focused on upskilling FDA staff in data related skills and harnessing the power of the incredible existing FDA data science community to drive data literacy and the joy that comes from working with and understanding data.He loves to spend his time talking FOSS (Python/R/whatever), Natural Language Processing, artificial Intelligence (AI), and all things data science. He also spends any night he can staring into the universe with his two dogs and his telescope.

________________________

β–Ί Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software

To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)

Thanks for hanging out with us! πŸ’›

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Welcome back to the Data Science Hangout. I'm Rachel. I lead Customer Marketing at Posit, and I'm so excited to have you joining us today. I was just saying today I'm calling in from Montreal, so a bit different. But the Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. We get together here every Thursday at the same time, same place. If this is your first time joining us, it is so nice to meet you. If it is anybody's first Data Science Hangout, we'd love to have you say hi in the chat so we can welcome you in as well. We're all dedicated to keeping this a friendly and welcoming space for everyone, and love hearing from you no matter your years of experience, your titles, industry, or languages that you work in. It's totally okay for you to just listen in here too.

Maybe you're out walking your dogs or in a coffee shop, but also awesome to be part of the party that happens in the Zoom chat. So I love seeing all the resources and links that are shared there too. There's also three ways that you can jump in and ask questions or provide your own perspective on certain topics. So first off, you can just raise your hand on Zoom, and I'll keep an eye out and call on you to jump in and ask your question. You could put questions in the Zoom chat and just put a little star next to it if it's something you want me to read instead. Otherwise, I'll just call on you to ask it. And then third, we have a Slido link where you can ask questions anonymously, and I'm sure someone from the Posit team will share that in the chat right now.

I do like to just add, we do have a LinkedIn group for the Hangouts. If you'd like to join that, it can often make it a bit easier to find some people that you might meet here in the chat too. And a quick note that if you are watching the recording at some point in the future and want to join us live, the link to add it to your calendar will be in the details below. There's no rule that anybody has to stay the whole time or come and go as it fits your schedule. But with that, I am so excited to be joined by my two co-hosts today, Kevin and Raju. Kevin is Associate Director of Nonclinical Informatics, and Raju is Data Scientist at the FDA. And I'd love to have you both maybe just kick things off with introducing yourselves, sharing a little bit about your role, but also something you like to do outside of work too.

Kevin's background

I'll go first. So I'll start with what I like to do outside of work. I think that's the first time ever. I don't want to forget. I really, let's see, I really like to play music. I play bass guitar, mandolin, guitar, learning banjo. All my kids, everyone has a ukulele in my family. The kids don't really play as much as I wish, but yeah, I think that's a big part of my life is playing music. And I'm into like jazz and bluegrass and Grateful Dead and all that kind of stuff.

So I think I wanted to give a little background, like kind of how I got to where I am, just to give context. So I got my doctorate in neuroscience and I was studying how stress affects cognition in rats. And I did a little bit of MATLAB programming and I was playing around with like this machine learning model to model their behavior, which I thought was a lot of fun, but my project really wasn't funded for that. I think I would have needed way more rats to like have the model converge, fitting their parameters and their data. So, but I really enjoyed that.

And then I did a short postdoc after grad school learning bioinformatics stuff, doing RNA-seq data analysis, where I learned how to program in R and Python and shell scripting. And then I got a job soon after at FDA, originally in CDRH and devices in vitro diagnostics for microbe, for infectious diseases, basically. So if I was still there, I would have been flooded with all the COVID applications for US, but luckily I ended up moving on to CDER shortly after. I had a little project there where we were trying to figure out how do you regulate next generation sequencing as an infectious disease diagnostic? So there's some kind of bioinformatics research stuff there.

But then I moved to CDER as a toxicology reviewer, so that reviewing animal toxicology studies that we conduct primarily before testing humans, although we continue to conduct them as we test longer durations in humans. And yeah, I was doing the therapeutic area of like pain and anesthesia and addiction. That was more neuroscience-y. I wanted to get back to that.

And right around the time that I joined as a toxicologist, the FDA was gearing up to start receiving structured data sets with these toxicology studies. Before that, it was just the study reports on PDF and actually paper not so long ago. But with getting data sets for these studies, that opens up a lot of possibilities of interesting things we can do with cross-study analysis to understand the background rates of findings or really look across large volumes of data and make better policy decisions and understand our policies better.

And so I started volunteering on projects to do whatever I could with these data sets because I had this background that most of the other toxicologists really had no need to know how to program because it was never really a thing in regulatory toxicology. And then eventually, I was lucky enough to get a position created for... I think actually I told my current boss at some point, I was just like, if I could do this stuff of volunteering full-time, I would do that. And then about a year later, she came to me and was like, I have an FTE position for you to do that because she was kind of under a lot of pressure. She basically runs the policy shop for the Farm Tax Review Program. And there's a lot of questions and policies that we have trouble answering because we don't have a great handle on our data. And so for the past four or five years, I've been in that position where I just do data science and informatics projects to support data-driven policy development for farm tax. And luckily, I also have some ORIZE fellows and postdocs where we do different research into developing different methods of working with this data, which then we can publish.

And I'm trying to grow not just the FDA, but with the industry as well, this open source consortium effort to utilize this data effectively and build standard tools because we have standard data.

Raju's background

Raju, do you want to go next? Yeah, thanks. So I'm Raju Rayavarapu, and I guess my main arc is going to be the data science work and stuff that I do is just sort of something that I discovered as like a passion and a love sort of thing. But to stick to Kevin's pattern here, I'm going to start with something fun, and then I'll give you guys my background here. So what I do for fun is in the last few years, especially since like, you know, the COVID stuff, I started to make a mission of like learning one new sort of life skill, as I've been calling them. So like a couple years ago, I learned how to like ride motorcycles, got my motorcycle license. But this year, I'm doing horseback riding, and I'm getting better at ice skating because as my partner, who's a fabulous ice skater, likes to point out, you know, using inertia to stop, like hitting the hitting like a wall to stop isn't like the best way to stop. So I've been like, you know, getting those skills up. But I'm a big fan of being outside and doing things like you'll be usually outside with my telescope if I can find time with my dogs to do that.

So my background is a little similar to Kevin, but a little different from Kevin. So I'm a PhD molecular biologist and biochemist. And so, you know, I did a lot of wet lab work, I spent most of my time, you know, looking at cell death mechanisms and looking at, you know, signaling pathways that govern cell death, particularly in like metastatic biology. And then from there, I went to St. Jude, where I worked on autophagic regulation and metabolic regulation and, you know, a little bit of autophagy for death. But while I was there, I got kind of interested in trying to figure out what the government and, you know, what industry and government and academia can do together, which led to a role at the FDA looking at public private partnerships.

And when I'd come into the agency, I'd done a little bit of scripting work. My dad's a computer programmer. So like, I was just kind of exposed to it at all times. But I never really wanted to do that. Because when I was a kid, I was like, I don't want to be like, I don't want to do the same thing my dad does, right? Like it's kind of you sometimes take that like hard line away. But I did like, you know, engage with some of these programmatic tools, I learned how to like code and visual basic a long time ago, so that I could play asteroids, like just really weird stuff like that. And then I used a little bit of R when I was working on like high throughput screening data to make better plots and all that fun stuff that you can't necessarily do with Excel.

But when I come to the FDA, I'd been just serendipitously sitting next to a friend who's a fabulous Python programmer, he's a semantic web expert. And I was explaining my project to him. It was like, hey, instead of doing it via like, you know, basic methodologies and using just like static, generating a static output. He's like, why don't you learn Python? It's not that you know, you'll pick it up. It's not that hard. He's like, it's kind of like a gateway towards a lot of other stuff. So I started dabbling with Python and R. And I started working on building dashboards and doing things like, you know, doing like web scraping and doing, you know, auto matching of concepts and terms and modeling and all this fun stuff for text.

From there, I ended up being picked up by the Center for Tobacco Products that had a lot of data that they were working with that was unstructured. And I spent a lot of time working on automation methodologies to structure their unstructured data. And while doing that, I was elected as co-chair of the Scientific Computing Board, along with my friend, Dr. Jamie Pettingill, who's also awesome, and I don't know if he's on the call.

And so Jamie and I get to essentially advocate on behalf of the scientific and research computing staff at the agency and their needs and, you know, what kinds of interesting technologies and tools that they want or have access to and ensuring that, you know, that is represented in a meaningful way. I also then was picked up by the Office of Data Analytics and Research, which is the FDA's chief data officer shop. And I'm a lead data scientist in there, helping establish some of their data science pipelines, some of their analytics pipelines. And I get to, you know, work with fabulous individuals. I just saw my colleague, Megan Merriman, pop into chat here, doing that, working on some of these analytics activities. And, you know, I get to just hang out with some really cool data science folk. I like to say that I kind of fell into this sort of field, but I was lucky enough to find some wonderful individuals that I get to work with.

Data science across the FDA

So, the main, like, one of the big driving forces at the FDA is that, you know, FDA is really pushing forward this model of IT strategic development and IT strategic modernization. And one of the big pieces of that is data literacy and data talent. And so, Kevin, myself, Alexis, and many of the people on the call, Tomas, we get to work in this space where we're working on data talent and data literacy, which is a driving force for, again, FDA's IT modernization plan and IT strategy, which, you know, you can look it up. It's out in the FDA.gov space. With that, I'm actually going to pass to Kevin to give some background on the data science community work that kind of spurred on many of the activities that myself and Kevin and a lot of us get to work on.

Yeah. So, I'm talking a little bit about the DSSD, data science and software development working group, which really started when Raju and I, we saw this talk, I forget the name of the guy. He was from the University of Buffalo Medical System. And he gave this awesome talk on how they were doing all this great informatics work, you know, and they seem to really have not just their data organized, but also like the infrastructure to support analysis. And me and Raju were talking to him and kind of just complaining about how difficult it was at FDA to do good data science work. And a lot of times you feel like you're doing it behind the back of IT, like they don't really want you to be doing this, but you're like, well, you know, this is actually good for the organization, so I'm going to do it anyway. But there wasn't a lot of guidance or a lot of support for doing that kind of work.

And, you know, we caught the ear of one of the more senior staff, this guy, Mark Walderhug, who's awesome in CBER. And he was kind enough to set up, kind of pull some strings behind the scenes and set up this cross-agency working group that he told me and Raju to then go ahead and lead. And so, we, you know, went through our emails and found all the people that we knew across, you know, we tried to get at least a

couple people from every center that was doing data science work. And we told them all, like, join this meeting and let's get together and start, you know, figuring out how to support each other.

And basically, the way we approached it, you know, in order to kind of bring people into the fold of really being actively involved, we decided to create three different workstreams so we could give leaders to each of those workstreams, a couple leaders to each of them. And then we had this kind of group of about nine people who were, like, core people that were invested. And so, we set up a community workstream, an infrastructure workstream, and a process workstream with the idea that, you know, community workstream would be an opportunity to, would focus on bringing people bringing people together. We, one, you know, the one thing we've always aspired to, we had a little trouble getting this consistently, is to have office hours, you know, where someone hosts it, or they, someone who knows something about some aspect of data science can host an office hours, and people can ask them questions, kind of like this type of venue. You know, we've also just, in general, had lots of presentations, opportunities for people to present their work and discuss it.

We did a hackathon at one point, which was really fun to, we formed some teams of people to try to solve problems together. The infrastructure workstream was something, actually, Raju spearheaded that, where we just tried to take a survey of what, what are all the platforms about, because the different centers at FDA kind of have siloed systems sometimes, and not everybody knows what's available to each other. And so, we're able to get a survey where everyone could say, you know, what platforms are you using, which HPC, and which center, which, you know, I don't know, just, you know, what's the IT support, or structures, or platforms that you use, and that was really helpful to get all that collated, and then we could kind of pass that off to, really, the, we started this group, like, the month that the chief data officer started at the agency, and so that was convenient, too, because he kind of was interested in what we were doing, and it was a good window for him into what the data science community was doing, and we've been able to pass off a lot of stuff from, a lot of the kind of grassroots stuff that we collect, or build, we're able to kind of pass off to his office to try to formalize, or, you know, make enterprise level, and, you know, and then the process workstream was, the idea was to build best practices. We had aspirations to build these documents of best practices, which we never really got around to, but we did do a lot of good trainings, like, we've got some good recordings on trainings on Git, and, you know, some basic R and Python stuff. Yeah, the documentation we found, I think there's so much good documentation on the internet already, it probably wasn't worth our time to create, and so we didn't. Hopefully, that's the reason we didn't, but, yeah, those are kind of the main things that we did, and we, yeah, there's one other project, one other project that we did. Sorry, my cell phone is acting up. I need to, it's messing up with my headphones. Okay,

the, we did this SWAT teams pilot, where we'd take a problem, like a real problem that somebody had in the DSSD, or someone they knew had, and we asked them to put it out there to the group, and ask for volunteers to come together and try to solve it, and we had, and yeah, and then Raj is going to talk more about how we've kind of passed this idea off to the chief data science office, the ODAR office, to formalize this, but this, you know, some examples of things, you know, during the COVID-19 pandemic, there was a group that was trying to organize data related to that, and we were able to provide some volunteers to support that, and another thing, I mean, this was like a simple thing, but I think it was a good example. There was a group in CFSAN, so the foods part of FDA, where they're tracking economic adulteration cases, and they had like this Excel, you know, they're using Excel to track all these cases, and they're like, this is probably not the best way to do it, and we were able to build a pretty simple, but effective, you know, R Shiny app, where you just throw it into a DT table, and put some UI stuff, so that you can use control terminology to, you know, select inputs, and things like that, so that when you populate this simple table, at least everything's, you know, you have your options that you're supposed to use very clearly designated. You can store it in a, you know, on a server, so that it's not like passed around between people's computers, you know, and that was the kind of thing.

I think that template of taking, somebody has some information in Excel, moving it to a DT table, and putting some UI around it, that's like a really common use case, where it ends up feeling a lot more powerful for the end user than what they had before, you know. Amazing how much time you can save somebody, or how much better you can make their workday by just giving them that tool.

Amazing how much time you can save somebody, or how much better you can make their workday by just giving them that tool.

I know Russ had a question in the chat about, like, who your customers are internally. Russ, do you want to jump in and expand on that one? Sure. I work in business, as opposed to the kind of stuff you people are doing, so I'm always customer focused, and so I'm curious, who is your customer, if not maybe the immediate, maybe down the road, is this for, like, a pharmaceutical, or a hospital, or who's your ultimate customer?

So, I was actually going to answer that question. I saw it, and I figured I'd just do it when I came off mic, right, off of mute here. So, our customer varies depending on, you know, the pipeline that we're either implementing or utilizing. Sometimes, you know, the customer is internal folk that have some sort of data-related issue that might be that, you know, to the people that are dunking on Excel in the chat here, right. Sometimes, they have large data sets that they can't handle because they are sort of using Excel, so we'll come in and help them kind of do some quick analysis, and that'll be like an internal one-off. We also collaborate with and work with across agencies, right, like we'll work with other agency partners, and so if they have a data need and we're the data repository for that, we'll, you know, work with them and generate either reports for them or produce data tables or data sheets that we can turn out, and realistically, it depends on the activity that we're tackling, who our customer ends up being, but if you need like a single word for it, it would be anybody within the FDA and any of our agency partners and sometimes external partners that require some sort of data assistance or data activity from us. Kevin works in, say, the Center for Drugs, and he does a lot of work for the Center for Drugs people that have data-related issues. I work at the Chief Data Officer shop, so we tend to work on some things that may be across centers, not to say that we don't focus on a specific center. So, I know that may not have been the most satisfying answer, but that's kind of, I think, a good way to think about it. Kevin?

Sure, yeah, and I would chime in that, like, yeah, I think a lot of the, in general, the FDA is obviously serving the American public, and a lot of our interactions are kind of customers are like the sponsors who are applying for approval of their, you know, clinical trials or their products, but a lot of the reviewers are kind of the end users who are interacting directly with sponsors and making regulatory decisions. Those aren't typically data science people, and so, yeah, what Roger's saying, you know, a lot of the data scientists at FDA are kind of serving those review regulatory people by feeding them data and tools to utilize data better so that they can, you know, support their customers outside better. And then I also, and I think we do try to do some public-facing projects as well, and like I was saying, you know, I'm very interested in developing open source methodologies for the, you know, the type of data that we review that sponsors send to us, you know, and in that sense, I'm hoping it's not even just a customer thing, that if I create some software, I'm hoping that we'll get contribution. And so far, you know, I've been able to participate in consortiums and get not just serve things to the sponsors, but to actually have a back and forth, you know, where we're building together.

Open source at the FDA

I wanted to ask you both this question because I've heard it come up quite a bit. It seems like there could be some misunderstanding across the community about support for open source within the FDA because I've heard a lot of people or a lot of people ask questions even in the Hangouts about using languages like R and Python in your submissions to the FDA, and I would just love to chat about that a little bit and see maybe what other follow-up.

Yeah, I mean, neither of us are in like the Biostats reviewer positions who, you know, those are the folks who really are, would be receiving those kinds of submissions. I think there is, you know, from my perspective, from what, you know, I probably should have said this at the very beginning, you know, as an FDA, I have to say the disclaimer, you know, these are my thoughts and views, nothing I say here, nothing I've said here represents the agency policy. But, you know, from my perspective, I think there's been a lot of infrastructure built around SAS in the industry and FDA, and changing that infrastructure over is, causes friction, you know, and there's not a, so there's, it's a process to, you know, things are built around expecting things to be they were, the way they were when they were set up, even the submission process, and so it takes some work to change that. I think a lot of people in FDA see the value of open source, but I think modernizing the review and submission processes, it just, it takes a lot of work to make sure that it, you can't break it while you're fixing it, you know, you have to keep it running, you have to keep the plane flying while you're building it, you know.

Yeah, and to add to that disclaimer, right, these are my opinions and thoughts as well, but the interesting thing is that is kind of a misconception. And, you know, and it could be that there was either, and here's how I'll tackle this one, right. I'm, I'm a big proponent of using open source technologies and tools, just much like Kevin, as well as many of my FDA colleagues that are on the call here, because, you know, it gives us flexibility, and it's honestly can be viewed as an industry standard technology platform. And so I'm going to like, split this question into an answer, one of the questions that somebody asked about how we're doing data science work and data science upskilling work at the agency. And so following up on the stuff that Kevin was talking about, with the data science and software development working group, and the SWOT teams, and all these things that we had worked on together, we were able to spin up a program in the office of the chief data officer called Data Forward. And if, and the Data Forward initiative is essentially trying to increase data literacy and data skill use, data skill tool usage at the agency. And what we actually focus on is because there is such a strong community of R users at the FDA, we actually give people the opportunity to learn how to use R and RStudio and Shiny to actually do some work. And that's not from any perspective other than we have people that use it, and it's something that we can get people spun up into fairly quickly. And the way that we do it is we will pull together either a project that is FDA relevant, or we'll pull together an FDA relevant data set. And we're piloting a new program where we're going to attempt to just do a self-learning with meetings with the FDA data science community to get your questions answered internally. And so that you can hash out what packages are the best or what methods are the best. But the idea being with this process, we give people exposure to things like Tidy, if that's something that they want to use, Tidy R and Tidyverse. And they become a little bit more capable at writing their own scripts to do their own data analytics and data analysis and can go back to their home offices and function as the data science advisors, mentors, or the actual scripters. But also it makes it easier for them to look out into the universe and see what the best tools that are being utilized by, whether it be academia, whether it be industry, are and become a little bit better at sort of bringing those technologies in and actually working with them. So that misconception is an interesting one, but as many of the people on this call can tell you, we use R, we use Python, we use the best tool for the job that we can find and we'll implement it as necessary.

Community building and cultural shift

So would you say it's been through this community building that you're starting to, I guess, remove some of the open source panic that may have normally or may have existed before?

Yeah, I think that there's both a cultural shift as well as like there is this idea of this open source community, right? I can't understate how much, rather I can't overstate and I don't want to make sure, I want to make sure it's not understated, that when I came into the agency, I got lucky and was sitting next to a very kind individual that was willing to teach me and take time to teach. And then I met people like Kevin, Alexis, Dave, like all these people that were willing to take the time to sort of like, you know, walk you through something. And so we started building that community and from that community we were able to show that, hey, actually, you know, there is a lot of this literacy that exists and there's a lot of interest to learn this stuff. And as more people start to use tools like R and Python and other open source technologies, it's sort of this like ground up sort of acceptance. And from a top-down perspective, there has been some, you know, cultural shifts and changes that have kind of helped push us forward. Bringing on a chief data officer was like, you know, a really big and helpful thing because, you know, he's very data and data is a data forward individual, right? Thinking from a data forward perspective. So, yeah.

Awesome. Thank you. Okay. Let me scroll through a few of the questions. I see, Eric, you just asked a question pretty recently in the chat. Do you want to jump in? Yeah. Thanks, Rachel. And I'm really happy to have you guys on. I've been involved in a few

of the R Consortium pilots for submissions and we've had great success up to this point. I will be transparent for those. One of the biggest challenges has been the infrastructure side of things where in the ideal situation, the development environment that we do on this pilot team, we could have a one-to-one kind of representation with what our FDA reviewers would have for these submissions. But we have learned through Paul Schutte and others that it's quite variable with people running things on their respective laptops and, you know, we have to have very detailed instructions on installing R and installing the R environment and getting this shiny bootstrap and all that. We're working on ways to make that easier, but I will admit, it does make the lack of a centralized type of infrastructure for the FDA reviewers to kind of hit at one central place a bit difficult for sponsors to navigate through that.

Obviously, it's not an easy-to-solve problem, but it sounds like you both have appetite for trying to help with the infrastructure side of things. Just curious what your perspectives are on how that can get better sometime, hopefully with enough traction in your organization.

Sure, yeah. Thanks, Eric. Good question. I think I want to preface by saying, kind of reiterating that, again, me and Raju are not really, like our day jobs aren't really in the biostats community of, you know, receiving these coded submissions, you know, the SAS code and potentially R code, hopefully more and more R code. But, you know, so, you know, like Paul Schutte, somebody like that would be a better, you know, representative to really field this question specifically.

I think what I can say, and I apologize for kind of redirecting, but I don't want to overstep my knowledge or expertise, you know, but I kind of want to redirect it to the stuff that we're doing, me and Raju, it's more enabling scientists who aren't statisticians, for the most part, to utilize open source to do their jobs more efficiently, to do their science better, or even admins, you know, even to do administrative things better. We're really, I think there's, from my perspective, there's, I guess there's less of a, there's less inertia to fight against when, you know, when there's, when you're working against, when you're putting a solution where nothing was in place than if you're changing the solution from SAS to R, you know, if you're talking to people who've never done any data science, there's so much low-hanging fruit, and there's, there's fairly little resistance to move in that direction, and so I think that's where our efforts end up flowing, because, you know, the statisticians, they, you know, they know how to do data, they know how to do statistics, and they know how to work with data, and they might have their ways that they've been doing it, and it's harder to change minds and reach people from that position than the lower-hanging fruit of people who are just like, oh my god, like, I can, I've been doing this thing in Excel, and I can save so much time, and I can get better insights, and I can generate, you know, I can automate, and I can do better visualization, and I can communicate better, you know, those are, those are more, those are the people who are more receptive, and where we find kind of more, more headway.

there's fairly little resistance to move in that direction, and so I think that's where our efforts end up flowing, because, you know, the statisticians, they, you know, they know how to do data, they know how to do statistics, and they know how to work with data, and they might have their ways that they've been doing it, and it's harder to change minds and reach people from that position than the lower-hanging fruit of people who are just like, oh my god, like, I can, I've been doing this thing in Excel, and I can save so much time, and I can get better insights, and I can generate, you know, I can automate, and I can do better visualization, and I can communicate better.

I don't know, Raju, did you add anything to that? No, I think that that's, you know, exactly it. I mean, we don't, again, Paul Schutte is a fabulous individual, and I was glad to hear his name come up, because he's, I mean, an incredible person that we get to work with on the day-to-day, but yeah, most of the stuff that we've been focusing on is attempting to enable those that have either little experience, or an experience in another piece of technology, and providing them with, you know, an additional tool set, and again, connectivity to this FDA data science network, so they can do things like, whether it be, you know, automating a daily task to actually being able to finally tackle a data project that they previously couldn't.

And the example that I use, and I use this example a lot when we're working through with the upskilling teams, is I was once posed a question, and it was on a small team with others, we, there were three of us, we were posed a question where a team thought that x was the outcome, and it was purely because they couldn't handle the mass amount of data that they had to essentially call, like, mine through to get to what they needed, but using a tool that, and again, I was using Python, so using Python, we were able to say, hey, actually, the answer is something else, purely because we were able to bring all that data together, and then show them that actually, like, we were able to correct their hypothesis, and, you know, from there, some of the people that were on that project have gone on to then be able to take that, and they actually, you know, shifted to R, because of the whole, you know, it was easier to get them R and RStudio at the agency, like, we were able to ship them to that, and now they do that analysis themselves, right? It was essentially an experiment in demonstrating what the value of these tools are like, and saying, hey, actually, this provides you with a little bit more capability than you had before. These were folk that would not traditionally have been exposed to these technologies, that now, when exposed, they kind of found some joy in it, and they moved forward with it. So, yeah.

Non-clinical toxicology and safety data

Yeah, I actually had one more thought I wanted to add. This is a little more CDER-specific, and probably still not satisfying what you wanted answered, but something from my position, right, I'm in, like, the non-clinical toxicology space, and it's another, it's a space where there hasn't been traditionally enormous biostat support, you know, and the nature of this data is exploratory. It's not powered, you know, we do, like, three to 10 animals per sex per group, and we measure, like, 100, 150 endpoints. You know, it's exploratory safety data, right? And so, we don't, the convenience, there's some convenient things from a data science perspective about that, is that we don't have pre-specified endpoints. We don't, we don't really have to worry, you know, you can do data exploration, and it's valid, because that's how the study was designed.

And so, the, but the one thing I can say, you know, as I kind of try to build more infrastructure and support for doing better data science in the non-clinical safety area, I think there is potential in the clinical safety area. You know, I've been looking into doing more kind of translational, like, can we organize our data better to understand, you know, the relationship between non-clinical and clinical safety findings, because they're really, in some ways, they're looking at different things, they're pulling different endpoints. But in general, my, from, again, I'm not a clinician, I'm not an MD, I don't do clinical data review, but from my, from what I've seen, the safety data analysis is a little bit more exploratory. It's not as pre-specified, and there, I think there's room there for more kind of data science innovation than maybe, whereas the efficacy endpoints, there's so much work, you know, that's been done, whether it was SAS or R, the work that goes into the statistical analysis of efficacy endpoints is really hard work. You know, like, people, like, there's not as, I don't think there's as much room for advancement as there is in the safety space. It seems like there's some low-hanging fruit. That's, I don't know, that's my two cents. Yeah, thank you very much.

Low-hanging fruit use cases

I know you gave a generic example of one of these, but I'm wondering, is there a favorite low-hanging fruit use case that comes to mind? Yeah, I mean, I think Kevin's kind of hit on it, right? Like, some of our data that comes in or we have stored in our repositories are either non-standard or need to be standardized in some way, right? And I never thought when I was, like, a 10-year-old that I would be, like, you know, geeking out over the fact that I could apply some sort of method to bring standardization and, you know, reduce the entropy of a dataset so that we could put it somewhere. Like, that wasn't, I want it to be, like, a race car driving soccer player, like every other kid on the planet, I think. But, like, so, like, there's a lot of those sorts of activities that are, they're low-hanging fruit in that they become, they allow us to enable downstream applications pretty significantly, like, you know, whether it be a already pre-existing dashboard that somebody's built that we can now standardize data into and bring a whole new dataset into it or whatever that may be. So, like, that's usually the example that I go to because that's, honestly, a lot of the work that I do is in wrangling and cleaning and making data standard so that it can fit into an existing pipeline. And then, you know, then we get to do the really fun stuff after that. Kevin?

Yeah, I think, I think, I don't know, like, Rachel, were you looking for, like, like a... I guess I was just, like, trying to look at, like, a specific example or, like, a person across the organization, or not, like, the specific person, but, like, what were they doing before and how did you make things easier for them? Yeah, I think, I think someone, I saw something in the chat where, you know, anyone who's using Excel in any organization, if they're, you know, if they're doing repetitive tasks in Excel, they, those are always low-hanging fruit examples where if, you know, if they have the time to get over the learning curve of using R or Python to do whatever they're trying to do, I'm sure they can do it better, you know, using better tools.

Right, sorry, I missed the specificity part of it, I guess. I guess a recent one was we got an external request for some data that was on the FDA.gov page, and the person that sent us the request was, I think, an economics grad student somewhere, and they had made this request, and they were, like, this, they're, they're, like, the way that the data is stored, which was stored as flat PDF files, they're, like, it's going to take us forever to get this data out, these data out, to actually do something with it, and so I checked with my supervisor, my, the CDO, I checked with my boss, and I was, like, hey, can I, like, I'm just going to do this for them real quick, and he's, like, okay, fine, and so I was able to convert those things from flat PDFs and send them back to them as CSVs, which is what they were looking for. They were asking for, essentially, database calls, and I was, like, well, I can extract these, convert them, and send them back, and in their note, I kind of found it a little shocking and surprising. They were, like, we want to do some, like, you know, complex analytics around this thing. They're, like, this will take us hundreds of hours to do, and I was, like, I can do this in, like, an hour, so we took care of that and moved that along, which, again, that's an answer to somebody, to Russ, who was asking about who our customers can be. They can be external partners, as well, and if we, if the, if the request comes in through a, a channel that we can handle, and it's something that I'm not saying everybody will get an answer in an hour, right, but it's, like, this was one that we could, you know, enable an external partner who could do some work to do some work. Yeah, absolutely, that's great.

The data science development program

I see, Nathan, you had asked a question a bit earlier about the data science development program. Do you want to jump in and ask that? Yes, sorry, I just, you all mentioned earlier, sort of, that I think there was some stuff on the website about your approach, or I'm just curious, as I'm with MedStar here in D.C., as we're undertaking, sort of, the same talent identification upskilling, if you guys have published anything, or any suggestions or thoughts as we, sort of, dig deep into our analyst pool and figure out how we take them on the same journey. Yeah, we, we presented about this at the ODT symposium, which happened in mid-December of 2023, and I'll see if that was a recorded process, but I can give you a quick breakdown of how we do it.

So, we started off with a pilot, and we actually used our existing data science community to reach out to individuals that they thought would be good members of the pilot to provide us with feedback on our process, but essentially, what we, excuse me, have them, have, have applicants do, we actually take an application pool, and we use the existing FDA data science community to review these applicants, and I think at the last, last application set, we had something like 200 or 168, or some, some, like, large number of people that had interest. So, we bring them in, we look at them, and one of the big things that we look for is, one of the big things that we look for is, we look to see what their, what their perceived value of these skills is for them to the specific organization or area that they are within the FDA, and anyone that can articulate that fairly clearly, we usually are, like, they move up the, move up the pool, because that's exactly what we're looking for, right? Not people that say, oh, well, I already do all this stuff, and I want to hop in with you all, which we're, at that point, we're like, you kind of don't need us. You're probably pretty, like, accomplished in finding, reading through package documentation or whatever it is.

So, we bring these folk in, and we have them kind of, we organize them into teams around a project with a mentor, and then before they even hop onto the project itself, we give them four weeks ahead of time, and we're experimenting with expanding that four-week piece out to do a standalone program, but we give them four weeks where I walk them through a basic dummy data set that I've created, and we farm it out to the data science communities and the mentors that we select for the program to then walk them through the basics of R, and then we hand them the project and say, okay, here's the specific project that we want you all to, you know, accomplish, and go forth as a small team with a mentor to guide you to learn.

And so that allows us to use the basic, what I like to think about is the basic applied science model of 70-20-10, right? 70% is experiential, where you actually have to put your fingers to the keys and have your code implode, because we want them to understand that it's not going to be something where they can just write a piece of code, and it's going to be magic every single time. Most of the time, they're going to spend their time wondering what this weird error is or how, why my data exploded. 20% of it is with relationships with their mentor, so we pick somebody that we find to be a champion for data science and somebody that we think could really guide people through their learning process, and their job is to essentially help them find answers, not necessarily give them answers, right? So, we do a 20% piece there, and 10% we give them formal, like, lessons and lectures, whether that be via, like, you know, the R platform, whether it be YouTube, whether it be Stack, like, I'll sometimes just send people, like, straight-up Stack overflow notes, being like, check this out, read this, and see if you can understand something in there. And we found that that's pretty

successful, and it's, you know, some people will go through it and realize they hate this, and they never want to do it again, and that's okay. We want them to at least attempt it, experience it, and realize that it's not for them. Some people will find that they enjoy it, but not enough to continue to script, but they become advocates for data and data-related activities at the agency, which is huge, right? They can come in, and if somebody says something, they'll be like, well, I tried that in this thing I did, and it didn't work out, and here's, like, you know, why I didn't think it worked out. And then we find some people that become the next generation, or the next, like, you know, cadre of advocates, who we then draw back in as mentors to teach the next team of people, and we're trying to build it as a self-feeding mechanism. Was that an answer to your question, to have asked it? Absolutely. We've been sort of trying to think around building the exact same thing, so it's helpful to hear that somebody else has gone through the process and at least has something that they feel like is working. Yeah, and they get a little portfolio piece out of it at the end, right? It's sort of something they can point to and be like, hey, look, I made that, and it's, like, it's kind of a very, like, fuzzy feeling, and we kind of, and again, I can't stress how wonderful the FDA data science community is, and all these people are always, like, I always tell the teams when they're working, I'm like, oh, yeah, like, you know, your mentors will hop in with a question, but we add many of the FDA data scientists into that, and it's, like, sometimes the answer is posted by somebody, or someone will be like, hey, let me help you with that. Let's hop off onto a side call before I even see the question, before some of the mentors even see the questions, which is, like, it's because this community wants to advocate and educate and build each other up, and it's kind of delightful.

to advocate and educate and build each other up, and it's kind of delightful.

R validation hub and open source challenges

I love hearing all these examples of community within organizations, and I see some comments in the chat about this sounding amazing and how can people apply to work there, too. I did also see in the chat there was some conversation around the R validation hub, and I thought that could be useful for the team commenting in the chat to share here. I don't know if, Anuja, you want to give a little bit of background on what that is. Sure, yeah, thank you. Yeah, we were just chatting about how sometimes with open source usage, specifically within a highly regulated space, there is additional challenges layered on top of that with regards to, you know, validation, what is validation, what is an adequate level of validation, and so I'll just quickly plug that R validation hub that I'm a part of. These are kind of, this is kind of our bread and butter and what we're trying to develop resources for and solutions for, and so if you're interested in attending our community meetings or just want to get in touch for an additional conversation, please connect with me, and I know that Raju and Kevin, you know, both of you are not specifically within the clinical review space, but I assume that as we move towards modernization, we'll be able to get in touch with you. So, yeah.

And I saw a couple of comments in here about, sorry, Diwani, I see that you have your hand up. Yeah, sorry, I was actually caught by surprise a little bit because I thought Anuja was going to talk about how actually using that application is harder to get approval for hosting open source and kind of do all those things, not the validation, because in my mind, it doesn't matter if you're using Excel or any other familiar tool, any process and methodology you end up using, whether it's SAS or whatever, still needs validation. So, validation is a uniform across, it's not dependent on the tool. Maybe there are some challenges by doing this in R versus something else. I would say Excel would be hardest to validate, which I think, based on some of the examples that were thrown in, it is so easy to change things in there, no source code repositories, anything else like that. So, yeah, I was just curious about that and wanted to bring that up, is like, I was very surprised that validation was the reason my OS was harder. I guess validation is necessary for any tool or software that you use to your point 20, but it's just that open source specifically poses additional challenges that can be a little tough to quantify in some instances.

I wanted to make sure we've re-shared that link as well, Anuja, if people are interested in finding it. I know there's so much great conversation happening in the chat there.

Rachel, I just wanted to note, because it might be missed on people, but you mentioned people are interested in applying for jobs. There's a data scientist position posting right now that's, I think it's not just FDA, it's like, I don't know, Rajvi, can you speak to it? What is it?

So, it's a government-wide data call for data science applications, so, you know, apply. It's, and essentially, it's a mechanism by which the agency is trying to bring in data talent so that then they can be utilized and, you know, brought into specific agencies after that process. So, it is a somewhat more, like, it's an intake portal out, and I believe it's open until, I saw one this morning that was open until the 24th of November, I think, of this year. Like, it's kind of long-standing, and I believe it might be rolling, and I'll look for some additional links and put them in chat for you all. Awesome, thank you. Tomas, I see your hand raised. Want to jump in here? Yeah, I just wanted to comment on the previous, the validation issue.

The validation debate

I think what we have to deal with is that the validation space is not rational, and very often we, if we call open source, if we didn't call it open source, if we called it something else, maybe there would be less issues with the validation. Because right now, we kind of delegate the validation to whoever the developer is, and, you know, I don't think that the SaaS developers are more reliable than, you know, the SaaS developers don't have emotional investment in the application, right? You very often can't get to them and all that. So, I think some of it is just behavioral, sort of behavioral science. We have to kind of change the narrative and make people understand where the validation truly is necessary. And then with Excel, that's, I think that's a great example. Like, you can, you know, Excel may be validated, but then a lot of the effort that you do in Excel is coming from the people's brain. So, what are you going to validate there, you know? So, by the way, the validation hub is doing excellent work, and I was involved for a little bit with, a couple years ago, and that was a really, that was a really interesting experience.

Thank you. I can't believe how, I always say this, but I can't believe how quickly this time is going by, and I know there's so many more questions too, but Eric, I see you have your hand raised. Want to jump in next? Yeah, just a quick kind of question back for TomΓ‘s. So, I definitely agree that when the software is actually getting developed, it's the same sort of effort that goes into open source for SaaS and things like that, but I wonder if one difference is with the SaaS being a company and everything, there's like the financial accountability if something goes wrong. So, someone you can, a company can kind of look towards if something goes wrong, and open source is missing that. So, maybe that's a distinction. I don't know if you think that's accurate. I guess it is. I just haven't, I think we don't have, we don't have enough examples of failure to really assess that. It's all sort of hypothetical.

Like, I've never heard of an R application failing in some substantial, you know, because it's kind of, it's propagated by sort of, it's like the Reddit upvotes. If a package gets successful enough, there's a reason for that. It's kind of an evolutionary reason, and sometimes that's not how the development works in software companies. You know, they look at the market and develop what they think could add value to their product. With the open source, it's the utility. It's some person really needs this and develops it to the T, you know. So, I think that's the irrationality of the space, that we don't really have good examples of failure.

If a package gets successful enough, there's a reason for that. It's kind of an evolutionary reason, and sometimes that's not how the development works in software companies.

But Samas, I just want to add, you know, I think for, you know, validation, like you were saying, you know, there are cases where validation is more important or less important, right, because the consequences are bigger or smaller, right? And that's where, I think that feeds into what we were saying before about, you know, like the efficacy, statistical analysis of efficacy of drugs, like that's like very critical. And that can come, it's a pre-specified plan, you know, and it can come down to like millions and billions of dollars on the line about how a p-value turns out, right? And those cases, like validation is obviously super important, right? You know, but in these other cases where it's like, I'm doing this task that I could do a thousand times faster as one step in a step with a thousand steps that all have the potential for some error to leak in, but like, that's life, you know, like those kinds of tasks, those kinds of things, those are the opportunities where we're having a lot more traction with data science because it's not, the questions about validation aren't as, you know, it's not, it's not like you were saying, it's not rational to apply high standards of validation to every single step we take, you know?

Yeah, and I'm going to kind of dovetail that a little bit into some of the questions in here about like, you know, package, like, I guess, validation of packages and how do we do it and all this fun stuff, right? And so it's actually a pretty fun debate that I get to have as part of the hat I wear from the scientific computing board for the agency where we get to talk to the, like the FDA IT infrastructure folks and the FDA IT, like your enterprise management folk about, you know, how we do one thing or another. And so like, you know, we always talk about, you know, community-based validation and management versus like, you know, security and the security folk doing their like review and management and all that fun stuff. And honestly, the best solution that we have right now is that the security team do validate those packages. And if they have any interesting questions, they reach out to the scientific computing board, myself or Jamie Pettingill, and then we're able to either route the question or route the answer and find the solution. So that is one of the activities that we do undertake. And it comes down to, you know, one, I can reach out to my friends in the FDA data science community to help provide additional insight as needed, but we have our IT InfoSec folk really paying attention and they are paying attention because they care about it as well, which is kind of awesome because they work with us to enable what we need. And if they find a problem or something, they're always willing to share it with us and then explain it to us, which I think is a huge cultural shift that at least I've experienced in the last, you know, however long I've been at the agency, which has been great.

Open source tools and recommendations

Awesome. Let me go quickly through a few questions here. I see Payam, you asked about some of the other open source tools. Do you want to jump in? Yes, thank you. Just quickly, what are some of the other tools or, I don't know, do you guys recommend at work for automation, dashboarding, infrastructure, anything?

Like I said, we use our Shiny for some of our projects. We also are looking into like, you know, Dash to do, you know, Python-based development and, you know, just Dash in general. One of the other tools that we use is KNIME, for those of you that have ever heard of or used KNIME, which has been around for a really long time. KNIME is another good one. Honestly, it comes down to like a day-to-day usage of whatever the best piece of technology is. Like if somebody wants to do, like if somebody wants to do OCR work, like, you know, they can either chase down a commercial product that exists within their specific organization or, you know, if they reach out to me, I'm like, hey, have you looked at Tesseract? Like, you know, again, it comes down to what the need is and what they essentially are trying to do with it. And then, you know, we have a whole slew of software and tools that have been essentially, you know, reviewed by the IT organization that we can then, you know, point them towards. And if it's something that's not there, then we talk to them about what it is, why they need it. And then, you know, we help them sort of navigate the process with our InfoSec folk as needed to either get it in or get an answer. Thanks. Is that list published somewhere on GitHub, for example?

No, our internal list is not published externally. But, you know, it's an interesting question and I can take that back and ask. Yeah, I would just add there's an aspect of convenience of like, you know, if we can do something using Microsoft Office tools like Power BI, because that's installed on everyone's computer, we should do that. R is easier to install at FDA because statisticians use it. There's, you know, so there's a user base for it. So there's already support for it, whereas Python is a little bit more neat, you know. But, you know, we're trying to build, but, you know, it's kind of that has to do with how, and I'm sure most organizations same way that, you know, whatever is already available to all your, you know, your organization, if you can do it with that, do it with that. And then, you know, try to advocate for, to fill the gaps you need. But it's not, so it's not saying R is better than Python, but it's easier to get installed at FDA. It's easier to use here. So we tend to lean towards it, especially for like new data scientists.

Career advice

Thank you both. Right as we end here, I want to zoom back out a little bit and just ask you both, I love asking this question. What is one of the most memorable pieces of career advice that you've both, you've either received or given throughout your professional journey? You want me to go first, Kevin? You got one?

I got one. I got one if you need time. All right. Yeah. I mean, I, for me, my career has been built on the old adage of don't ask for permission, beg for forgiveness, where, you know, I just, I do things that, you know, like if you see the opportunity to do something that's going to add value to your organization or your coworkers or the problem you're trying to solve, like don't

ask for permission first, because it's going to, it takes longer to get permission than it does to just do it. And once you do it and you show people, Hey, I did this thing and it's really valuable. They usually just say, thank you. And we're, you know, it's easier to explain it after you did it than to try to explain and get permission. And that's, that's been my, yeah, that's been helpful for me. Love it. Thank you. That's a really good one. And, you know, I'm going to shout out to a grad student that I worked with Dr. Callie Versagli, if she ever watches this video.

But I mean, this is a piece of advice I've gotten a lot in my life, right? But this one time really stuck with me where if you see something that you find is a need, and it's something that causes you grief or frustration, it's okay to roll up your sleeves and use, you know, Kevin's model of like, you know, ask for forgiveness or whatever you want to do, but roll up your sleeves and actually attempt to make a change, right. Rather than just complaining about it. So, you know, I've, most of the jobs that I've had, and most of the tasks that I've gotten at the FDA have been because I'm like, here's a need. How do we resolve that need? Here's a potential solution to that need. And let's see if it works, right? Taking that experimental biologist hat and taking it to like, real world problems and actually doing it. So if you see a need, and you can find either someone that will help you advocate for it, or you can find some way to solve that problem, or at least help somebody solve that problem, you should just lean in and work with it within the bounds of, you know, what a human being can accomplish. And don't burn yourself out and all that fun stuff. But, you know, get your, just attempt to make a change, I guess.

My other two cents motto is all science is data science. All science is data science. All data science, right? Even grocery lists.

My other two cents motto is all science is data science. All science is data science. All data science, right? Even grocery lists.

Thank you both so much. I love Alexis has been the chat as well to find your community. And just want to say thank you all so much for being here and for spending time with us in this community. And if this was anybody's first time and you want to join us back next Thursday, we are here every Thursday, same time, same place. I'll put the short link in the chat where you can add it to your calendar. But Raju and Kevin, really appreciate you joining us here and sharing your experience. This has been great. Yeah, thanks for inviting us. This was a lot of fun.

Thank you so much. Oh, quick plug I forgot to do. The call for talks for the POSIT conference in August this year is now open. And so I put a blog post in the chat as well. And hope to see many submissions from you all here because we'd love to see talks at the conference as well. Have a great rest of the day, everybody.

Transcribed by https://otter.ai