image: thumbnail.jpg
Novo Nordisk’s Journey to an R based FDA Submission
Join Novo Nordisk, leading pharmaceutical company in diabetes, obesity and other chronic diseases, as we celebrate reaching our NDA milestone, having recently submitted our first submission package featuring outputs exclusively written in R, along with select ADaM datasets also written in R. In this webinar, we will provide you with an in-depth look at our computing platform and our internal R package suite, including insights into their development. We will reflect on the process and setup we have in place and provide you with a sneak peek into what the future holds for us. You'll get an exclusive look at our journey and learn how we overcame challenges, as well as how we continue to innovate and streamline our processes. Don't miss this opportunity to gain insight into our approach to NDA submissions.
Speakers:
Ari Siggaard Knoph (MSc):
Ari is a Principal Statistical Programmer and International Lead Programmer at Novo Nordisk with a seat on the pharmaverse council. He is a driving force behind the adoption of R in clinical deliverables and submissions at Novo Nordisk. Ari is also the author and maintainer of multiple R packages and leads the Novo Nordisk pharmaverse team.
Steffen Falgreen Larsen (MSc, PhD):
Steffen is a Statistical Specialist with over 8 years of experience in driving the shift towards R and Python at Novo Nordisk. He has a master's degree in mathematics with a specialization in life sciences and a PhD in biostatistics and informatics. Steffen has designed and co-developed a Statistical Computing Environment for R and the internally developed package ecosystem. He is a seasoned R user and developer.
Anders Ellern Bilgrau (MSc, PhD):
Anders is a Statistical Specialist at Novo Nordisk and an Adjunct Associate Professor at the Department of Mathematical Sciences, Aalborg University. He holds a master's degree in applied mathematics and a PhD in biostatistics and informatics. Anders is not only a skilled statistician but also an experienced software developer, with a special focus on R development for many years. He has played a key role in co-developing the R platform and internal packages used in Biostatistics at Novo Nordisk.
Transcript#
This transcript was generated automatically and may contain errors.
the Director of Life Sciences and Healthcare here at Posit. Very excited to bring to you today the ninth installment of our Life Sciences Series. Previously, we have highlighted Roche, BMS, Merck, GSK, Novartis, Johnson & Johnson, and others. And you can find these Posit webinars linked below.
And today we have an exciting webinar featuring the work of Novo Nordisk. We're going to highlight the journey to an R based FDA submission. And we'll kick off the presentation today with Ari, the Principal Statistical Programmer. We'll then pass it over to Stephan, Statistician. And then we'll end with Anders, Statistical Specialist.
It's such an exciting time for open source drug development. From 2017 to 2020, we had important things that had happened. R and Pharma, the R validation hub, the Pharmaverse. And then into 2021, we had an R pilot submission to the FDA. And now a little over a year later, we have a full submission in R by Novo.
And it's so exciting to see the pharmas are migrating to an open source backbone for their clinical reporting. And what sets Novo apart is the depth for which they're tackling this effort. The change management from a multilingual statistical computing environment to the package management and validation and to the end user support and education. For example, they even have their own internal R podcast. And all the while contributing to community efforts like R and Pharma, the Pharmaverse, the pilot submissions. And so it's with great joy to bring you this session today. And now I will pass it over to Ari.
Thank you, Phil. We're so excited to be here and to be able to tell our story of all the ups and downs we've experienced in the past few years.
Before we begin, I just have to say that this presentation represents the view of the office and not necessarily New Orleans. And any of the regulatory communication that we'll show here will not be applicable to all projects. So please communicate with the appropriate health authorities about your submission.
The first R-based submission
Without further ado then, earlier this year, we submitted our first R-based submission to FDA, PMDA and other health authorities around the world. And this happened early in 2023. And we have multiple learnings and feedback from authorities that we will share later on in the presentation as well.
Incidentally, this year Novo Nordisk is actually celebrating its 100th anniversary. And Novo Nordisk started in 1923 as Nordic Insulin Laboratory. And it was at a later merger with Novo Industry that the current name Novo Nordisk was formed. Here in Novo Nordisk, we focused on treatments for people living with serious chronic diseases. And we have a strong history in diabetes and now in obesity for which you may know the company for.
In biostatistics where we three sit, we work with the first three steps shown here. So we work with the clinical trials, data and evidence generation and approval activities. In the evidence that we submit to authorities, we include STDM and Adam datasets that follow industry standards such as the CDISC. We submit programs, documentation, analysis, tables, figures, listings, you name it. And what we're going to share today is our story of getting to do all of this with R.
Going back to the beginning
Now going back to the beginning, then there's one thing we really, really want to highlight and one thing that permeates everything in this presentation. And we put this quote on the slide, that the goal is the transformation process that is revolutionary in result, but evolutionary in execution.
that the goal is the transformation process that is revolutionary in result, but evolutionary in execution.
Or in other words, it's more a matter of taking steps rather than thinking about how big those steps should be. Now, back in 2015, we had a lot of people Now back in 2015, we all joined the biostatistics in Novo to an environment where everything was dominated by the SAS programming language.
We loved R and we quickly noticed a huge R shaped hole at Novo. And to our surprise, Gibbs and version control systems were also absent and we were not the first to notice this here. So we got R installed on our local computers and we used it every now and then for these one-off non-critical tasks.
In 2017, 2016, we started to introduce R to management and our stakeholders and colleagues for non-GXP related work, where we were showcasing Shiny as an incredible powerful framework for data visualization that you just couldn't get anywhere else. We had a Shiny app for data exploration, the bottom one you see here, and one for assistance in pharmacokinetic and pharmacodynamic data reviews.
We were also using R for crawling SAS logs, comparing stuff and checking stuff. And development of this continued through 2017. And in 2018, we continued to push for adopting some use of R. And the collection of R programs we were using had grown so much that we now combined it into a single package named NNR. And this package was the first thing we put in version control and we also set up pipelines for testing it.
Commitment to report a trial in R
Now, come 2019, where we finally saw a commitment from above to report a trial in another language than SAS. I remember receiving this strategy email and Stephan was sitting at the desk right next to mine. And I showed him this and we literally ran into our manager and said, if anybody's going to do this, it has to be us. And it wouldn't take long until we got the nod to go ahead for this. And we both thought, shit, it's real.
How do we even do this? We don't have a TXP environment for R. We don't have a system for packages. I mean, we don't even have a strategy for packages. So we get down to this and I'll hand it over to Stephan to tell you more.
Yes. So at this point, we are still using R from all local laptops without any system basically at all. So we were in dire need of a statistical computing environment.
And actually our initial thoughts regarding this new environment were in terms of stakeholders and the change management of those. So we wanted to produce table figures and listings that were identical to those that were already produced in SAS. Because this mean that there would be no change for the stakeholders regarding the language that we would choose. It would also mean that we could use the existing workflows that we had set up for producing reports that we're going to send to authorities. And we could on the same trial work on both R and SAS.
We also wanted a system that were hassle-free for the end user. So we were trying to cater to our colleagues that were used to programming SAS. So we really wanted a server that they didn't need to think so much about packages, environment as such, but more about getting to the actual programming. So it should also be so that if you've got a code snippet from your colleague, you should be pretty sure that that would also work on your computer, given that you were using the same version of R.
And then finally, with this new SCE, we really wanted to highlight version control. So remember at this time, we weren't using a version control basically at all. We had a few things that we had it on, but not for anything regarding trials. So here we wanted to produce an easy way for initiating a version control for both trial and collaboration work, such as packages.
Pillars of the infrastructure
We quickly saw that these four things would be the pillars of our infrastructure. We really wanted an IDE based on RStudio workflow or the opposite, Workbench, because it was such a great IDE and familiar to so many R users. We did, however, want to make it the server-based solution so that you would simply log in to the server and be sure that you were in common ground with everybody else. Then we wanted to use Git for our version control because that was so familiar with most of the R programmers as we had used it in the past. And it will also come with great features, such as feature branching for our development stuff.
We wanted to use Azure DevOps, both for project management, so for organizing tasks, but also for all of the code repositories that we could share those. And of course, for CICD pipelines. And finally, but definitely not least, we wanted to use deposit package manager for all things regarding R packages. We wanted to use it to share our internal packages. And we also wanted to use it for creating subset of packages that we will call approved packages for use.
Now, when we started working on this SCE, we were hit pretty hard by the number of things you need to consider for making a GXP compliant system. I won't go through all of these now, but suffice it to say that there is a lot to think about. And so instead of starting from scratch, we took a look at what do we already have. I mean, at the time we were able to produce all of the stuff that we needed for the reports that were going to the authorities. So instead of starting from scratch, we wanted to piggyback on what we already had.
So we wanted to launch a computer next to the one we already had. Now, the one we already had were located somewhere in Copenhagen, having a file share with all of the data and output and such. So we wanted to put it next to it, except we didn't really want to put it next to it. We wanted to put it in the cloud because we had no idea how many users were going to be on the system. There'll probably be a few in the beginning and then a lot as time progressed. And luckily we got a really, really good cloud architect to help us out on this.
And together we quickly saw that in order to fill all of these GXP requirements, we needed full infrastructure as code. And we needed to have version control on that code so that we could see who changed what and when. We also, while I quickly identified the need of machine images. So for all of our compute instances, they should be built on a machine image so that we could spin it up and be sure we would get the same result the next time as we did before. And finally, as we didn't want to do in a manual task, we wanted to use release pipelines for releasing the entire infrastructure in different environments.
Building the GXP environment
So in 2020, we released our first GXP and classified environment for doing R stuff. It would look something like this where a user would come in from the workbench. Here they would have access to the RStudio IDE. They could program in R and they would have access to all of the packages they would need from the Posit package manager. They could see all of their repositories and get those from the Azure DevOps. And they could install their code on a home directory, which was a file share in the cloud. Now we still had all of our data residing on our old computer on the file share, which were located somewhere in Copenhagen.
In terms of features, we didn't do much the first couple of years, but we did make a larger machines. Then we make more workbench machines and a load balancer in front. Because of the growing need for computation in regards to the users. And then come 2022, and we finally started to introduce new features. We introduced Python on the workbench server and PyPy on the Posit package manager. We introduced new IDEs, so that the programmer would be able to select the IDE he or she was most comfortable with. And we also introduced a lot of new features that the IDE he or she was most comfortable with.
Then we introduced a caching system. And this was because the file share located in Copenhagen had an insanely slow connection to the cloud. So this way we would only have to wait for a dataset to be transferred once, and then everybody else could use the data from the cache.
We also released a Posit Connect instance. Mostly for sharing our markdown documents and running Shiny applications. Now this was built on the same image as the workbench machine, which meant that if it worked on the workbench machine, it would almost certainly also work on the Connect instance.
But we did have one problem that was still remaining here. And that were some users, regardless of how big we tried to make these workbench machine, would hog up all of the resources that were available. So in 2023, we finally released a high performance compute cluster in the type of a Kubernetes cluster, where a user could spin up a Docker image.
And through that, they would get the entire IDE, but they would be ensured that they would not step on the toes of anybody else, any of the other users.
And speaking of users, let's just take a look at the user uptake on this system. So here we have depicted the number of users that we have on the Y-axis and the timeline on the X-axis.
And we can see that we didn't have that many users in the beginning as we expected. These users were mainly those that were assigned to the trials that were going to produce results that were going to use R for that.
But as you can see, the time grows, we can see many, many, many more users. And in the end, we're actually not only supporting the biostatistics department, but also users from other departments within Novo Nordisk.
And I thought we would have had such a great success in the user uptake on the system, if we didn't have a really great strategy on handling packages. And now, Anders will take us through the handling of these packages from CRAN.
Package management strategy
Yeah. Thank you for that, Stephan. So the NN package management really started in 2020 with the... Sorry.
Remember that our organization was used to SAS and SAS only. So introducing a new language that is very much unlike SAS, that's challenging. So SAS is nice and simple. You need to install SAS 9.4. It's 10 years old. It has been patched a couple of times, but it's still in use. R on the other hand, that's not as simple.
R is evolving at a higher pace. And R also has all of these packages. And each of these packages, they come with their own versions, they come at different times, they have their own life.
So these packages are what makes R great and powerful, but there is no free lunch.
So these packages are what makes R great and powerful, but there is no free lunch.
So dependencies between packages, they quickly become too complex if you just pick and choose your versions arbitrarily. You need, you'll quickly create a set of packages that does not work as intended together. So we wanted to create an environment that is sufficiently fixed and stable, but still flexible. And we opted for this shared baseline model as it has now become known.
So here you lock each version of R to a particular date and time, and you install packages in versions that were available at that given date. So we do this so that we know that any pair of dependent packages, they were tested at that point in time, and they were tested that they work together. So that means that, for example, our R version 4.02, that's locked to some date in September, 2020, that we will always get dplyr 1.0.2, despite how many releases of dplyr that has been since.
So teams simply just choose their version of R. They can deviate from the standard ones that we have in the system, but if they do so, they must use renv. In general, we actually recommend that you always use renv.
Risk assessment for contributed packages
So what packages can and should we use? Well, they can be roughly categorized in this way. So they are the base and recommended packages. Those are the ones that ship with R. They are low risk, and they are very widely used. The contributed ones are the one we need to do a risk assessment on. So of these, there are the widely used ones. They are low risk by virtue of the enormous user base.
There is the well-tested and documented ones. They may be more niche, but they are still considered low risk. It is the risk that pose a problem.
So when we do a risk assessment, we lean heavily on the R Validation Hub's recommendations on how to categorize. So the main points of categorization is the type of use, the maintenance, the popularity, and so on. So at the end of the day, this is just a human risk assessment, and the way we prompt this assessment to get made is by a Shiny app.
So our users can, in the IDE via an add-in, they will be able to select a requester app, and that simply starts up. So in the app, they can write what package they desire and the reason for it, and they can also see which R version and version of the package this will be available in.
So when they submit this request, it gets version controlled in Git and gets shipped. That kicks off a pipeline and then risk metrics. The risk metric page tests or performs risk metrics for the package in each version of R that we have and that associated library, and in all of the environments that we have on the system.
Manual, sorry, a reviewer can then manually inspect these results and arrive at an informed decision. If the reviewer chooses to approve this, then it will get fed to our system code and then released upon the next release of that system.
Internal R packages
So we now have a GXP environment and we have a strategy for packages, but we still do not have R code for producing tables, figures, and listings. We do not have batch execution and logging capabilities in R, what about those?
This is also handled by the packages, our internal packages, and these internal packages, they started as a single package, one package to rule them all in 2018, and this was developed on a local laptop, and it was simply a gathering of all the usual functions that have been made.
In 2020, we introduced the nnBiostat package that was simply defined as the accepted packages. So users would write library in nnBiostat and get the most useful packages.
In 2021, we realized that nnR had become too big and it was split into packages related to specific tasks. So nnR then became just a simple umbrella of all of these packages, like you know from Tidyverse.
So the most important of our internal packages is nnAccess, that is for getting data without worrying about paths or data formats. It's nnPlot for creating NovoStyle plots. It's nnTable, which defines the grammar of tables for laying out tables and creating text-based output. So we wanted to mimic what we already did in SAS. It can also produce HTML outputs. We also have nnExport for exporting table figures and lists that fit into our production pipeline. And we have nnLog for batch execution and logging. Here we actually use niraspin to create nice-looking logs.
So, and although these packages, they fit together in a complicated network, each package is just like in the real world, it has its own repo. It is managed in an open-source way internally. And we have version numbers, a maintainer group, we have pipelines, and we have boards for managing bugs, features, ideas, and much more.
So we aimed to follow a common and good software principles of modularity, and we tried to build up functionality like Lego bricks. So we tried to segregate the responsibility of the packages and define appropriate extraction levels. So at the bottom here, you see the packages that are close to the system. They are system-dependent, if you will. And on top, you see the packages that are close to the task at hand.
At the top, users shouldn't worry about how the data is stored, where is it stored, and so on. They should just concentrate on the task.
So for example, a programmer wishes to use nntfl to make a mean plot that will call nnAccess to fetch ADAM data through a connection set up by nnRemote. And after the plot has been made, nnExport will take care of exporting the data to an appropriate data format and associated files that we need.
So this has the benefit that we can change underlying architectures in the system and simply just do small changes in the lower-level packages, if you will, and the users won't see a difference. So this is also the reason that we could inject a caching system into the system.
Tool chain and code quality
So let's talk about the tool chain that we use in Novo. So we use git for version control and Azure Pipelines for executing code upon code changes. We use useless for the description file and the news file. We use package down for generating a website of documentation, and we use our oxygen to write the documentation.
We use lifecycle to convey the stability or the lifecycle of a function within the packages. And we use test.uncover to test the packages. We also use our markdown for writing longer format documentation in the form of vignettes. And we use DevTools to do the normal system, sorry, the normal package development.
So upon a code change, we will test again the package in a matrix way. So we test against all our libraries, all our versions of R and across all of our system environments.
So we know that if, for example, an upcoming change in development to the system will break something in our packages. We then build the package and then we release it. So in the process of releasing, we are updating the package down website and we ship it to the internal repo in the package manager.
So we realized some time ago, but we implemented it this year that we need the ability to deprecate our versions. So say we no longer want to support the oldest version of R we have and stop, then we have made version specific repos so we can stop updating that. This also allows for hotfixes if system changes are needed.
So we try to ensure a high level of code quality by enforcing a main branch policy. Nobody can change the main branch directly. Only pull requests to the main branch are accepted. The pipeline needs to run with success and no errors. A feature or bot needs to be linked to the pull request so we know why we have to change. And we need at least one maintainer to approve to ensure a code review.
What it looks like in the real world
So now we both have a system and we both have the capabilities to execute R code. And I think I'll hand it over to Stefan so he can show you what it looks like in the real world.
Yes. Thank you. So I would like to show you how a script usually looks when we do it in Novo Nordisk. So all programs we have, we'll start with the header as I'm showing you now.
This header will include a short description of what the program is going to do. So in this case, we're going to derive summary statistics for antibody levels grouped by treatment and gender.
We will have a very small outline just indicating the most important steps of the program, making it easier to read for another programmer.
Then the actual code will almost always start with a call to library and then biostat. This will ensure that we have the packages that are most typically used in trial conduct loaded into the system and that they're loaded in the same order for every program.
And coming to the main part of the code, we will almost always use NnAccess because that's how we get access to the data. We can call NnAccess on a specific trial that will give us a trial access object.
A trial access object will have access to all of the trial collections that are supported for the selected trial. In this case, that could be, for instance, ADAM or SDGM. Here we're accessing ADAM data by calling db.adam and then we can say what dataset we want to extract.
And notice here that we do not have any paths here. That is taken care of by NnAccess. And this means we can actually share this code snippet without thinking about where it's actually executed. It will always get the same dataset. It also allowed us to inject this caching system so that every time a user gets a dataset that has already been loaded, we'll simply use the cache and avoid long waiting time.
So when having the data, we will usually use normal dplyr calculations in order to derive the statistics that we want to use. So for instance, here, we will filter the data. We will summarize the statistics that we want to put in the table. And then we can, in this case, group them by treatment, gender and the parameter code of interest.
This will give us all of the data points that we would like to include in our table. However, now we had this goal that we wanted to produce tables
that were identical to those produced in SAS so that we could be used in our usual pipelines for producing reports to authorities.
However, we were using this text-based format which has the tables that would span multiple pages and we had no support for that in R. So we created our own package in a table for doing that.
And here you can initiate a table rather easily. You simply state what data is this table going to use and then you can say what columns are going to be in this table. You can do some light formatting saying that we would like mean and standard deviation to be displayed as such, or we would like minimum and maximum to be displayed with a semicolon between.
This will produce a table looking like this, but we wouldn't be quite happy yet. Now, normally we would have all of the summary statistics nested under each other.
So we would have to make the table a bit better, but you can note that we have the display that we wanted and we said that we wanted in the call.
In order to nest the summary statistics underneath a new variable, we use the functionality called addTransLong where we can say we want these columns nested under a new column called sum.
And we can also add some formatting, for instance, to say how many decimals do we want to apply for the different numbers.
That will produce a table like this, which is a bit closer to what we usually have in our reports. However, you can see the two new columns down the bottom, the sum and the name column. Now, we would usually have the sum column nested under treatment and perhaps on the gender. And the type of summary statistics, which is the name column, would go under the parameter code.
So we could use this by having these two added lines of code where we have addTransLine where we can say summary column should be nested under treatment, which again should be nested under gender.
Similar, we can add the stop of the table saying we want a name nested under the parameter code. That will produce a table like this, which looks like something we've seen before that we are going to do in Novo Nordisk.
We are able to produce other types of tables using this. So we can export this via FlexTable to, for instance, PowerPoint or a Word document or directly to HTML.
But we are focusing today on what do we do when we want to export it to a simple file. And here we use the nnExport functionality from the nnExport package. Here we can simply say the output name that we want to export and a file name that we want to export it for.
Then it will figure out the rest. And then we're actually done with a script that can be used to produce such a table.
Running scripts and log inspection
And in order to run it, we will use a function called runScript that comes from nnLog. This function can take both R documents and R Markdown documents, and it can execute those in batch and in parallel.
And it will give you some sort of information about how the execution actually went. In this case, we can see that the execution actually had an error. So let's inspect that log.
As you can see here, we get a nice HTML log, and that log is produced via the spin functionality from the NetApp package.
We can see that we have this header on the top that just gives a short description of how did the execution actually go. Then we have the actual description and the outline that we put in top of our document. And we can see here they are nicely formatted. After that comes all of the code chunks that we have.
We will also see all of the outputs in the log. And in this case, we saw that we had an error. It will also highlight both errors and warnings so that they're very easy to see as you scroll through the log.
And finally, in the end of these logs, we always have an expanded session info. Now, this session info will include both session information about the packages that we had in the session, but also packages that were simply available at the system. It will also include options that were set in the environment and other environment info.
But that is actually how we execute scripts and how we do them in Novo Nordisk in general. But I'll hand it over to Anders. And Anders will tell us a bit about how we got our colleagues to actually do this.
User support and education
Yeah, thank you. Yeah, so once again, the way we have primarily been doing our user support and our education is to let everybody know that there's help to be made. So the help is here.
So we have primarily been supporting people via a one-stop shop website that collects all of these package down sites.
And recently we have introduced Stack Overflow for Teams as a Q&A. Until then, we have been using Microsoft Teams, we have been using DevOps, we have been using plain old mail to do this support.
In terms of education, we have promoted self-learning and learning by doing. This has been done through a podcast, as Phil mentioned. And we have internal e-learning as well. This is built on top of degree.com.
We also provide lots of external references and e-learning to the outside world, the ones you can find on the web. We have done courses and demos on how to get started with R and Novo, but it's really the website that has been doing the heavy lifting.
So the website looks like this, and we call it the R.doc homepage. You simply write R-doc in the browser when you're on the corporate network and you'll be taken to this.
So it's a one point of entry homepage for everything. We have the motto that everything should be here. If you cannot find it here, then it should be here and we will make it.
So this site collects all the documentation and references just mentioned. It has the podcast, it has the Pakistan website. It has vignettes, long format documentation. It has an entire output catalog so you can see how a program is structured. It has lots of guides and more.
We also link further on to all of these e-learning and education sites.
Something that you all know that is present in R, this is not actually a given in other languages where you can simply write help in the IDE and get the documentation. So we, of course, also convey this to our users. And we really try to write high quality documentation on our packages.
So you can get this right away. We've also tried to do the equivalent of use this in a package called nninit, and this will help you get started the first time you log in. It will help you troubleshoot if you have any problems. It will help you initialize and set up stuff, trials, repos, new packages, just in general, initiating stuff.
And we have also tried to integrate and utilize all the RStudio features with add-ins and snippets to make them a nicer user experience.
The podcast is 10 to 15 minutes of how-to's of everything from the complex to the very simple. How do you get started? How do you make your first output? That sort of stuff. And again, anybody can contribute here.
Stack Overflow for Teams
The one thing I want to highlight is this Stack Overflow for Teams. We have just recently rolled this out. It's only a month old, but we can tell already now that it's going to be really good.
We have already 174 questions and 255, sorry, not 55, 25 answers in there and 65 daily active users. So we really had no good place for Q&A, and we wanted to avoid answering the same questions again and again.
So this site really helps spread the support burden. It helps build you a knowledge base, and it's not just for code. And this is also familiar to all programmers. So there's no education for this needed.
So if we have any regrets in terms of change management, it's just that we didn't get this out sooner.
So if we have any regrets in terms of change management, it's just that we didn't get this out sooner.
Submission timeline and milestones
So now I'll hand it back to Ari, and he'll tell you about the submission.
Now, getting back to what I highlighted in the beginning, we've always believed in the power of taking steps rather than thinking about how big they should be. And to put everything in ANOS and CIFN into perspective and how you could do all of this while doing a drop project, we'll just look at a quick timeline of the development here.
So in 2019, we got the commitment to report a trial in another language than CES. In the beginning of 2020, we had collected functionalities into two R packages. Soon after that, we initiated the first R program on this trial, chosen to be the first one reported in another language than CES.
And we even did that before we had the first workbench in production. Unfortunately, the trial got delayed, but we were determined to prove that we could use R in our deliverables. And we did an experiment with learning a DSUR instead.
Later on in 2020, we split up the functionalities and created packages and package shelf of everything that was to come. And we were working on the documentation and released the RDoC homepage in the end of 2020.
And from doing this DSUR, we just learned an incredible amount of stuff during our first deliverables. So we updated a lot of core functionalities and the packages. We update the workbench once more with load balancing and multiple servers.
Soon after that, I managed to convince our management that we're now ready to run phase three trials in R, and we do the first commits on these trials. Immediately after that, we follow up with five clinical pharmacology trials. And finally, right before Christmas in the end of 2021, we report our first trial ever in another language than SAS.
And between 2022 and 2023, we just have an incredible amounts of first happening. And we also update the workbench once more because we're sending so much data in and out of it now that it crashes when we run these big add-on programs.
So what we end up with is a huge package of R-based GFL programs and some add-on programs in R. But basically after we proved in the end of 2020 that it was possible to deliver a DSUR that followed basically all the same processes internally as we do for clinical trials, we were using R in all project deliverables from that point on.
Submission strategy and FDA approach
One thing to note is that we initially did not intend to submit all of those R-TFL programs. If we look at FDA guidance, then our plan was to submit all of our add-on programs, including those made in R, and then having those R-TFL programs internally and instead submit our SAS parallel programs for our primary and secondary efficacy analysis, which is something that we had done in projects before.
And that leaves us with this gray area that we were still creating a strategy for. I'll get back to this in just a second.
The add-on R programs that we submitted, they were designed with minimal dependencies on internal packages, as we weren't sure how to best submit packages at this point in time. We did use our internal package in an access for getting access to data, but we didn't plan on submitting it just as we didn't submit SAS-based macros that only work internally.
We did still think that we needed to tell authorities about abuse of R. So you can see here an excerpt from a type C meeting package where we mentioned that we plan on submitting R programs and we will specify R-related stuff in the ADHE and submit internal packages if they are not publicly available.
In our pre-BLA meeting package, we get more specific and we mentioned that we intend to use the R package, package light, which packages R packages to TXT in order to pass them through the submission gateway at the FDA.
And this initial approach is the method used by the R Consortium Working Group R for Submission in their first pilot submission to FDA. We also mirror their approach in the ADHE where we add something about R in the standard section and we add an additional very short appendix on what you would need in order to run the other programs.
So with this, we bundle everything together and we submit our first multilingual submission package to FDA.
But there's still this gray area that we need to tackle because we know that PMDA has already made pre-submission requests for additional TFL programs. And we know that it's not unusual for FDA or other health authorities to ask for additional programs.
And lo and behold, it doesn't take long until we receive this.
An information request from FDA stating, to facilitate the efficiency of the safety review, submit all programming codes used to create the tables and figures in the ISS and the safety sections of the CSRs.
And now we're thinking, Chip, it's real. We really need to figure this out and we only have 10 days to do it.
You can imagine the stressed looks on our faces and a lot of deep breaths that need to be taken at this point in time. But after we have calmed ourselves down a bit, we get to it.
Responding to the FDA information request
So we knew all trials have been executed with the renv setup. So we could use the renv log files to look up versions of all of the internal packages.
We then use PackageLite to take the package sources and repackage them all into a single PackageLite file for each trial.
In the submission, we now included the PackageLite file, the renv log file package to text and modify to use MRAN for public packages. We include renv's activate script that bootstraps renv. And then a script that would unpack source packages, set up a seller for renv, rebuild the packages and initialize an R project with the renv setup and within a dedicated folder structure such that all of the internal Novo packages could run.
In the ADHE, we update the appendix to include some information on these new files that we're submitting and which commands were needed to restore the environment and how you could modify the NN access function in order to make it run in a local R project. So we bundle all of this together and ship a whole bunch of R-based TFL programs together with the R packages and these additional files to the FDA.
And again, I mean, at this point in time, we're on top of the world. I mean, we did it. Now we also submitted packages and even within this deadline.
But FDA returns again and they can't get it to work. And now we're really thinking, shit. And together with that message, they also include the following in the information request. Submit R code for all safety analysis with as minimal package dependency as possible and no dependencies to R packages developed internally by Novo Nordisk.
And now we're really thinking, shit. At this point in time, we're doubting if we can get it to work. At this point in time, we're doubting everything that you heard up until now in this presentation. The package strategy, the submission strategy, the future of R at Novo Nordisk. We poured so much energy into making this a reality and now we're at this pivotal moment where we make it or we break it.
We poured so much energy into making this a reality and now we're at this pivotal moment where we make it or we break it.
They can't restart the environment and they want us to reprogram all the programs. So what do we do?
We do something we very, very rarely do. We invite FDA for an informal meeting.
In the meantime, we go back looking at our build script and optimize it once more. We also optimize the R and V log file to use Posit's public package manager as at this point in time, MRAN has now shut down.
We write a very long and very detailed instruction with screenshots of everything that's happening and what you should expect to see when you're restoring the environment, when you're executing different parts of the scripts. And we ship this off to FDA before the meeting, hoping that we now have done it right.
The meeting with FDA
Now this meeting happened quite recently and minutes before the meeting, we are quite nervous. Like what are they gonna say? Are we going to be grilled?
And the meeting starts and it turns out they now have been successful in restoring the environment. And we were just thinking, yes. And they bring up the programs and mentioned that part of the reason for them asking is that it gives FDA an opportunity to see how code inside and outside of packages are built. And this would be a great learning opportunity for FDA.
The statistical revenues in the meeting with FDA are just so super nice. And we realized one very important thing from this meeting. Submissions in R is new to us, but it's definitely new to FDA as well.
Submissions in R is new to us, but it's definitely new to FDA as well.
Looking to the future
And this brings us to the future where we have three main points that we wanna dive deeper into. We wanna have a stronger R dedication. We want improvements to recreating environments outside of the in-and-outs. And we wanna fully embrace the open source.
As part of the R dedication, we have multiple fully R-based projects already started and now also dedicated teams to support our package and system development.
We have learned some very valuable environment lessons around the rebuilding across operating systems. And the fact that the official package manager stored binaries of packages was really invaluable in this submission as we didn't need to compile from a source on Windows machines.
And we already have many ideas for how we can work tests around this into our workflows and also guide colleagues and peers for regulatory interaction. And last but not least, fully embrace open source.
The cleanup of all of our internal packages will start soon and we will move towards open development on GitHub. We will rebuild a lot of our core functionalities such that it will work for the broader programming community and aim to get our packages onto CRAN and dive much deeper into FarmReverse.
Before we end, I really wanna acknowledge all the package contributors and system contributors that we have, the programmers and statisticians that fought to learn R on these trials throughout a very hectic time, the invaluable input and knowledge coming from our peers at other companies working with our working groups under our consortium, the R validation hub and all the efforts going into making something like this possible.
We would also like to acknowledge the fact that FDA agreed to meet with us and advertise in the end of the day. It's also just nice, normal people working at FDA. They are much more open and welcoming of R than we anticipated. And we encourage that you reach out just as we did. So thank you so much for listening to our presentation.
Q&A
Awesome. Thank you, Novo team. That was fantastic. Gosh, what a aid this is gonna be for other pharmas going down this path.
We have more questions than I think we have time for. So we're gonna tackle a few of them. And they also came in at different times before you had got to that point later on in the presentation. So if you think we've already tackled it, just let me know. But I'm gonna go based off of popularity and we've got 30 plus questions. So let's tackle a few of them.
And so let's start off with this first one. How can we ensure that R code, including internal R packages is responsible, is reproducible on a reviewer's machine, such as that of the FDA?
That is a very good question. Would you take it Stan?
Yes, so that is a good question. And of course we cannot completely ensure it. What we can ensure is that we can reproduce the environment such that they have the same packages available, the same version of R available, that they, in that way we can make sure that they have data in the places we would like them to have data in. And that way use our programs.
And that was what we did in the scripts that we sent to them now, was to actually from the file that we have submitted in this M5 folder structure, make it possible to recreate the entire folder structure that we normally work upon, and then reuse the in an access facility and in a biostat functionality to ensure that the packages that they have loaded is the same as what we are seeing.
And thereby be very close to a fact that they will get the same result as us, as close as we can get.
Fantastic. Anybody else on the NOVA team want to mention, comment?
No, I think that's sufficient. Yeah, I think that's enough. All right, so let's go to another question here. So the next question says, how long before it becomes a common practice in NOVA to use R for trial related programming?
So I think, well, so I mean, I think doing this one drug project at first has really kind of kicked off like an internal revolution for doing this because we proved that it was possible to do it. It's been hard and difficult and there's a lot of change management involved, but the benefits we've seen from this, like reusing TFL objects into result meeting slides, like all the benefits we get from doing HTML based stuff now is just spilling over into other projects. I think probably we're closing in on maybe 30, 40% of the trials now are starting up in R.
And we have, I think 115 trials ongoing currently. So the uptake is quite big.
Fantastic. Any other comments? Let's jump to the next one. All right, so the next one is about the scope of the undertaking. How many personnel are needed to maintain the complexity of this environment?
Yeah, so that's a hard question also because at NOVA we noticed this has really been a bottom up adventure where we saw from the top that we would like us to make this report or trial in another language than R. And then we were pretty much allowed to run with it.
So we've been a few people working on the packages. I would guess six that would do usual commits. We have been mainly three to work on the statistical computing environment. In terms of the actual development of it, we had way more in terms of getting it GXP validated because that's just a much, much bigger adventure. But in terms of active development.
And then how many for actual trial conduct? I mean, there's been a very steep learning curve on the trial. So I would say there's anything between 30, 40, 50 programmers and statisticians that have been running all of these trials, creating R programs. I must say all the trials have been running in parallel, which has also made the workload quite a lot bigger. But a lot of people have been involved in doing actual trial conduct.
And one thing I think we need to add is also that as we do it internally open source, lots of people step up and contribute. The podcast, for example, is somebody came up with that and we said, go with it. Let's put it on this RDocs thing. The same with this DeGreed. Another guy came up with that. And so we just grabbed those balls and we really try to encourage our users and our friends to do that sort of thing.
So it's hard to put in terms of FTEs because we can't, we don't know actually. And on that point, that's what's been a learning for us who have been developing the packages first to actually start to let go and let the community be the main drivers. So for instance, for this table package that we have, we're getting more and more contributors around the company for features that they would like it to include. And also feature requests, of course, that's why can't I use the DT package directly from this? So we will of course start to implement that so that it has full support for that as well.
Interesting. So I've got a couple more questions. So I'm gonna go a little bit faster on these because we've got a lot of people still on. So let's keep going. So this next one said,
what can you clarify by what you mean by first R-based submission?
It says, are all the programs in code were in R? Yeah, so all of the tables, figures and listings that we've made have all been in R.
So at Novo, we have been using a data set for analysis results for quite some time, meaning that the statistical analysis are still done in SAS, put into a huge data set and then reported through our TFL programs. So statistical analysis are still onto the side, onto and into SAS, but everything else in regards to TFL have been developed in R. And then as well, we have some of these R-based atom programs, which have primarily been for the data set where we just need the additional compute power that we can get from being in the cloud because they have been extremely huge.
Business benefits of moving to R
Awesome, another question. What were the business benefits of an R solution relative to commercials or for example, SAS?
That's a good question. I guess, so like Stefan said, this spills over or I believe it's UI. It spills over in all sorts of other things. So when we do presentation slides or we do other stuff, it's much more easy to pick up and do something more powerful with these shiny apps and so on. They also can utilize the packages so we can kind of, again, build up functionality.
So it really spills over, but again, it's impossible to say, like in terms of money, how much again it is, but we see it kind of starting to permeate into other non-GXP tasks and other stuff that we do.
And one other very important aspect from the business side has been how do we get more employees involved in biostatistics? Because we are doing more and more trials, so we need more and more employees. And here we have seen that there are just so many more people we can get who know R than there are actually people coming out who know SAS. So from a business benefits, I would say one of the highest values is getting new employees and getting them up to speed fast.
I would say one of the highest values is getting new employees and getting them up to speed fast.
Python support and multilingual environments
So you just mentioned research and other areas. And for the late stage here, a question that comes up, I think a lot of people are wondering about is do you plan to use Python in a submission too? And does your scientific or statistical computing environment support Python and R development and apps?
We have Python on the system. It is an experimental feature, if you will. So, and as far as I know, we don't have any plans for doing Python-based submissions, but we feel that there is a pressure. So the users or some users really want that as well.
So now we are trying to come up with similar strategies for handling Python so that, so you can really have a true multilingual system. But we have seen, we have started the process, but it is a bit harder with Python than R because we don't have CRAN, where we have a much, much better control. And we know that the packages are tested together. So we can have this nice strategy for how we put the packages in, where we know that these packages are going to function together. That doesn't really apply to Python. So that is at least the problem that we are tackling right now in order to do it.
R environment update cadence
Awesome. How about one more question here? How often do you update your different R environments?
Yeah, so we've been doing, I guess if we even them out, it would be like a half yearly update. But it has depended on how big are the features that we're working on. Quite recently, we had a large gap between them because we were working on this Kubernetes cluster. So that was sort of a big thing. We needed to develop the Kubernetes cluster and also get it DXP approved before we could release it to the broader community. So there we had, I think, almost a year. But usually it would be about every half yearly.
And then we have a commitment, both in terms of user request, but also for ourself, that we really want to go down to every second month to have updates of the system so that new packages can get to users much, much faster than they do now.
And also, I mean, if we're diving more into open source, where the release cycle, at least from the stuff we see in farmer versus is quite fast, like we need to be up to speed if we want to include like the latest updates into our trials as well. So we really want to get the release cycle down.
Exactly. But one thing though, even though we have a rather strict package management strategy, we have made it so that all users can actually, if they work a little bit for it, not much, but just change the repository, the options repost argument, then they can lock themselves into a newer version of the packages and start working from there. So it is possible. And we do have some projects that are starting to do that if they feel that the system has fallen a bit too much behind that we need to work with the newer packages available, and we need to make sure that the new package is available, then we have shown them, okay, how can you do that yourself and still ensure that you are compliant?
Lessons for the next R-based submission
So my team just mentioned that we still have a lot of people on. So if you have some time, let's keep going. What the heck, right? The questions keep flooding in. How about this one? I like this. So for your next R-based submission, what are some things you might do differently that jump out to you?
I think one thing is, I mean, we realized now that a statistical reviewer at FDA or another health authorities might not have adequate time to kind of get acquainted with a whole set of packages that's being submitted. Meaning that if you're building like super smart functionality in a lot of packages and they all like really work well together, getting that kind of thrown in the face as a reviewer might be too much and you wouldn't have time to dive deeper into it.
So we're thinking of a lot of ways of how do we expose like the derivations, the calculations, the analysis that we're doing in programs, either through like better logs or kind of surfacing more of like the deep based derivations that we're doing. And then, I mean, we want to do a submission with the Admiral as well. We want to use more of these farm roast packages in them.
And definitely, as I mentioned here, testing in a Windows-based operating system is something we will definitely be building into kind of our submission pipeline that it is something that just needs to work.
IT and Biostatistics collaboration
Fantastic. All right. How about two more and then I officially will end the webinar for today. So how about given the significant overlap between the responsibilities and functions of IT and Biostats, how was collaboration between these two sectors? Facilitated?
Definitely. Yes. And there is actually a good example of this. We, at the very first time, the IT development sort of developed an R server without so much response from Biostatistics. And in that server, we said, you can go here and you can start programming. We said, you can go here and you can start programming. And the first thing I tried was, okay, so I want to install Tidyverse so that I could get going.
That won't work here. We don't have all of these things installed in the system so that you can actually do that. And okay. So I can't even use Tidyverse. This is going to be a steep climb if we're going to use Base R for everything.
So from that day, we started a really, really close collaboration because we sort of saw the need that even though you know everything about IT, you might not know all the needs that there are for an R-based system. So from there, we would have both R developers and IT developers working closely together, ensuring that we would have the best system possible for the end user.
Experience with regulatory agencies beyond the FDA
Okay, last question. This is it, I promise. How about, you know, one thing we hear a lot about are other regulatory agencies besides the FDA for submissions. So someone says, can you share about your experience with R submissions outside of the FDA, EMA, PMDA, et cetera?
Yeah, so I really want to bring up an example from the R Consortium working group, R for Submission, where we also tried to reach out to PMDA through some of our peers in the Japanese Pharmaceutical Manufacturing Association, JPMA. And the message we got back from PMDA was that they didn't feel a need for a pilot submission because they already accepted our submissions.
And what we have still included in our correspondences with the PMDA, for instance, that we're using R for different stuff, we're giving them flow charts of stuff. And with PMDA, we even suggested that we would submit RDS files instead of XPT files, just because we had these huge datasets that were more than 250 gigabytes in XPT, which I think any local laptop would be difficult to open those up.
And they've been very open. And I would say from all the health authorities that we have talked with, none of them are questioning the use of R. They are still coming with the same scientific questions about the submission. Of course, here we got some technical questions about restoring the environment, but it was because they wanted to make it work. So only positive things, I would say.
none of them are questioning the use of R. They are still coming with the same scientific questions about the submission.
Well, fantastic. Thank you again. I think this has been one heck of a webinar and the community is gonna really enjoy watching this afterwards. I suspect we also can tackle some of these other questions maybe in a blog post. So thank you so much, Ari, Steph, and Anders. Truly wonderful to hear about your experience, the journey you went down, and the collaboration with the community and internally has been fantastic. So with that, we do thank you one last time and thank you to the community that's turned out for the webinar today. And stay tuned. We'll be posting this online here soon for those of you that may have missed some parts. And hopefully we see you at R and Pharma coming up in October. And with that, we'll bid everybody farewell and have a wonderful afternoon. Thank you, everybody.