image: thumbnail.jpg
David Granjon & Bo Wang @ Novartis | User-friendly, self-serve tools | Data Science Hangout
We learned how their team designs production ready apps for clinical trials, from wireframing activities to automated deployments.
How do you get feedback from users when you don’t hear from them directly?
To know whether the applications are properly used, they use a tracker within Posit Connect. This is basically an application David developed using the Connect API.
With the Connect API, he can see “who is accessing the application, which gives you some insights like which team is using it and how to target support.” If you develop an application for several teams, you can have some insight of which team is using the application the most. If you see one team is using the application 20 times more than another one, maybe you want to invest more people to provide two paths here.
He supplemented this information by developing an R package, shinyHeatmap, to record in-app usage. “Each click is recorded to identify dead zones to refactor design,” he says. “If you have an app no one visits, maybe it’s poorly designed.”
For more information on that Connect API: https://pkgs.rstudio.com/connectapi/
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to the Data Science Hangout. Hope everybody's having a great week and are enjoying the Shiny conference. Thank you so much to the Appsilon team for putting on this amazing conference and for asking for the Data Science Hangout to be a part of that. It's really cool to be included in the conference.
If somebody is here from the Hangout and doesn't know what I'm talking about with the Shiny conference, can we share the link in the chat so they know how to join in too? But I know we have quite a few people here joining the Data Science Hangout for the very first time. So thank you all so much for joining us.
I'm Rachel Dempsey and I lead our pro community at Posit. So we host this Data Science Hangout every Thursday at the same time, same place. So what is it? What's going on here? This is our open space to chat about data science leadership, questions you're facing, getting to hear about what's going on in the world of data across different industries. And so every week we feature a different data science leader as my co-host to help lead our discussion and answer questions from you all. There's no presentations or slides, just all conversation with everybody.
We're all dedicated to making this a welcoming environment for everyone. And so we always love to hear from everybody, no matter your level of experience or area of work. This is a bit different than the Hopin platform that you're in for the Shiny conference. So I just want to make people aware of that to be a bit careful with unmuting and those things that come with Zoom. We'll be watching out for it too, but just wanted to make that clear.
It is totally, totally OK to just listen in here. But there's also three ways that you can jump in so you can ask questions or provide your own perspective on certain topics by raising your hand here on Zoom. If you're not familiar with Zoom, it's in the bar below where it says reactions. If you click on that, you can raise your hand. You can also put questions in the Zoom chat. And if you want, just put a little star next to it if you don't want to jump in and read it yourself. I'll know that maybe you're in a coffee shop or something and you want me to read it.
And then lastly, we also have a Slido link where you can ask questions anonymously. And our team here at Posit will share that into the chat in just a second. We'll keep posting it in there so that people can find it as well. It's great to see all the messages in the chat. We do share the recordings of each session to the Posit YouTube and our data science hangout site. So you can always go back and rewatch or share with a friend.
But with all of that, thank you so much for joining. It's so amazing to see this big group that we have here today. And thank you so much, David, for joining us as our featured leader. David Grandjohn is a senior full stack Shiny developer at Novartis and the founder and maintainer of the open source R interface organization where he develops Shiny extensions like BS4- and Shiny Mobile. We also have Bo Wang, senior expert data science at Novartis, who's also joining us in the audience today. He'll be helping with any questions that come up on their data monitoring committees at Novartis as well.
Introductions
So let me bring you both here to introduce yourselves as well. David, I'd love to have you introduce yourself and maybe share a little bit about your role and also something you like to do outside of work.
Yeah, sure. Hi, everyone. Thanks for having me. I'm David Grandjohn. I was born in France 32 years ago. I'm married. I have one wonderful daughter, a wonderful wife. I'm doing data science at Novartis since four years. My background is about applied mathematics. So I have quite a sort of Swiss knife background where I can do physics. I can do mathematics, chemistry, a lot of different things and also background in web development. So when I learned web development, I learned HTML, CSS, PHP and also database on the side.
I'm super glad I was able to attend the keynote before from Colleen because I like sport a lot as well. I hate running, but I like cycling a lot and probably for the same reasons as Colleen. So, yeah, at some point, no matter what sport you are doing, if it's making you better, feeling you better, having new ideas, taking some insight of what you are doing, that's just what you have to do.
We did have a really fun picture of David with his like goggles on, like getting ready to go out for a bike ride that we might have to go up with the recording. Thank you again, David. Bo, so nice to have you here, too. It would be great to have you introduce yourself as well.
Thank you. Thank you for having me. First of all, I love running and I hate biking. I didn't know that about you, David, that you didn't like running. Maybe we can team up as a relay team in the triathlon or something. I hated biking when I tried it.
Anyway, so my name is Bo. I work on the same team as David and Novartis. We are data scientists who build tools, pipelines to support our clinical statisticians and other data science functions. My background is in biostatistics. Since I started working in the pharma industry, I've been working on more on the tool development, almost like software development side than actual trial analysis. Although I have been working on supporting data monitoring committees, as Rachel mentioned before, and there seems to be a bit of interest in that on that topic in previous meetings. So that's why I'm here. Outside of work, like I said, I like running. I like traveling. I'm actually right now in Memphis, Tennessee, even though I'm normally based in Boston. I do not have a wife or a daughter. I do have a dog who I treat as my daughter. She's at my foot, napping. And that's pretty much it. Glad to be here. Thank you.
Building great user interfaces
Thank you so much for joining us. And just a quick reminder to everybody, if you're not talking, if you want to just mute yourself, it will be really helpful.
OK, cool. Thank you so much. So I get the pleasure of being able to ask some of the first questions as we wait for questions to come in. But if anybody did just join us after the initial introductions, I wanted to let you know that you can ask questions a few different ways. So you can put questions into the Zoom chat. You can also raise your hand on on Zoom and I'll call you in. And we also have a Slido link which we'll share again where you can ask questions anonymously, too.
But something I was really excited to start off with asking is, David, I know you think a lot about building great user interfaces. And I was curious to hear tips you have for all of us in communicating with the business through data visualizations.
Yeah, sure. So from I think we started a big project last year, which is around probability of success. And then for this project, we really thought about another way to start our Shiny-R app project, especially, you know, from the user interface perspective. So I would say when I joined Novartis at the beginning, most of the time we were, you know, rushing to code. We developed the interface, the app, and then we showed the app to the end user and then they are not happy with the result. We completely changed the process. And with this project, we basically start with wireframing. So we met with the user, we drew the user requirements, we discussed together for one week, two weeks, and then we present some mockups, some wireframing. So you could basically either use a paper and a pen or you can use more sophisticated tools. I would say it's a bit of pity that most of the good wireframing tools you have to pay. We are using Balsamiq, Novartis. You may use it, you may not. It depends if you know. And then based on that, we spend around, I would say, one month before we can come to an agreement and then we can start to code.
Overview of projects at Novartis
Awesome. Thank you. I'm starting to see a lot of questions coming in the chat. Thanks so much to the Posit team for helping me organize these. But Libby, I see you asked a question. Do you want to jump in?
Yeah, I was just wondering if we could get an overview of some of the big projects that you've been working on at Novartis. I know that Novartis is a pharma company and that it's really big, but being a person outside of the pharma universe, I sometimes don't know what you guys are working on.
You want to start on this one, Bo? And then I will take next.
Sure. So two of the bigger projects I've been working on, one I've been working on for a couple of years. Basically, when a clinical trial is being conducted, there are a lot of data analysis on the patient data that we collect from the enrollees. And for later stage, for later, there are like earlier phases trials in a small amount of patients with lower doses. And then once the safety established, you move on to later phases with larger number of patients and higher doses for efficacy analysis, for example. But the patient safety is always monitored through all the phases.
And for later stage studies, the sponsor in PharmaLingo, the sponsor is a pharmaceutical company that pays for the clinical trial. Anyone inside the sponsor company are not supposed to know the actual treatment of the patients, given if the study is a randomized controlled trial. So we need to hire external partners. We need to hire we as sponsors have to hire external experts to be unblinded to the actual treatment code and review safety data to make sure that all the patients in the ongoing trial are still safe. So one of the big projects I've been working on in the past couple of years is writing and managing an app, a Shiny-R app that has a JavaScript backend to display some of the safety data in the more user friendly way than, say, 200 pages of PDF.
Because the DMC members tend to be very highly regarded, very expensive, their time is very expensive, experts in their field. So they have very limited time that they can give us to review the data. So having an app as a supplementary tool aids their review of the data with the limited time that they have.
So having an app as a supplementary tool aids their review of the data with the limited time that they have.
Since I joined Novartis, I've been very lucky regarding all the projects I received. Surprisingly, at the beginning, like during my first month, I was given a project, not about clinical data, but for human resources data. Because you have to imagine, you know, Novartis, it's like 100,000 people working. And then you have many departments. It's not only clinical, but it's also sales, human resources and many other things. And basically, we had to develop a tool to better manage the time people spend on projects. And this tool was basically first developed with Shiny-R, with a lot of databases around just to make sure that data are persistent. And then later, the tool was improved using other technologies like React. But in the end, yeah, it shows, you know, how you can integrate Shiny-R in the whole ecosystem. And that was a great project.
Then I joined the Probability of Success initiative, which was really a rewarding project because we really had a checklist and tried to respect all possible best practices. This checklist was based on the mistakes we made on other projects. And then, yeah, we started with the wireframing. Then we had like a strong CI-CD setup. So yeah, I'm responsible of the CI-CD setup of the team. So making sure we have... So we are using GitLab, so we have runners with proper installation, system dependencies, everything.
So if you want to do like headless testing for your Shiny apps, you make sure you got the proper installation. Yeah, so setting up with runners, yeah, how to manage the release, you know, the production release.
So this afternoon, we had, during the conference, we had the production release for POS 1.0.0, the first one, which is like one year of work. So that's quite a big achievement.
And also, I've been involved... So POS is clinical data, right? It's really to... I didn't really give the background, but, you know, when you work in pharma, pharma companies, they invest billions into making a treatment. It's not just like tens of thousands, it's really billions. So you need to make sure what you are going to do is successful at some point. You need to know to analyze the market on everything. And these probability of success tools provide like a data-driven solution to have an improved diagnosis for your probability of success.
Migration and infrastructure
And yeah, finally, I've been involved in migration. So it's more like in the server infrastructure. So to migrate from Shiny Server Pro to RStudio Connect. So initially, yeah, we had Shiny Server Pro and now we have RStudio Connect. So I had to supervise this migration. So essentially talking about thousands of applications and make sure, you know, those applications work correctly on RStudio Connect, POSIT Connect now.
So that was quite challenging because all apps were not necessarily following all the best practices, you know, essentially around package dependencies. So sometimes you are not able to find, you know, where a package is coming from. Yeah, and the application was just not working anymore. So we did a lot of work around RL to set up reproducible environments for many projects, which has a learning curve. But yeah, we are making progress, right?
Yeah, and we are also tracking usage on RStudio Connect. So using the API, getting the data from the server to see, you know, how people are using the applications and stuff like that. So that's very, not only people data, that's all data, all possible data.
That's awesome. Cool to hear you're using that API too to get those advanced usage metrics. I'm seeing a ton of questions coming in here too. And I want to make sure I get to the ones from the chat too. And so Sep, I see you had asked a question around validation. Do you want to jump in and ask that?
Validation in regulated industries
Sure. Thanks, guys, for jumping in and sharing your stories. I know Bo already kind of answered it, but validation is usually, especially in kind of regulated industries, is an arduous task generally. And so I was kind of really interested to see, is that something that you guys have built that required validation to be undertaken? You know, sometimes people have alternatives to that. So if you could speak to that, I guess, a little bit more, Bo or David, that'd be great.
Yes, I think arduous is the right word to describe the whole process, as I discovered in the past couple of years. I came from my previous experience in my work throughout my graduate school career. I had never encountered the issue of having to validate a piece of software or a piece of application. Not in the like code review and testing sense, but in the process documentation sense. So it was definitely a learning curve, and we have had a few iterations of processes that are based on software validation.
When I say software, you can think of RStudio, the Shiny server, or the RSConnect as a piece of software that we have existing processes written, authored by IT to support. But apparently those processes are a little, there's a little, there's a little too much overhead for a Shiny application.
So in the past, I would say year and a half, two years, we've been working with our QA colleagues, our IT colleagues, and our colleague who are managing our deployment system or analytical system to kind of communicate both ways. What does quality mean for our QA colleagues? What does quality mean for IT? What does quality mean for business? And try to reach an understanding of how much testing, how much documentation is needed.
What does quality mean for our QA colleagues? What does quality mean for IT? What does quality mean for business? And try to reach an understanding of how much testing, how much documentation is needed.
It's still an ongoing process. We've had a few pilot projects deploy, deploying apps for use in DMC, but there are like exceptions for certain studies that does not apply to other studies. So the solution for certain issues don't really translate to long-term solutions that you can write in the process, in like a business process on how to validate a Shiny application. So that's the part that we're still trying to figure out as we incorporate more different types of studies in the program.
Travis, I see you had a question as well. And I forgot to say, I know there's a lot of people who are here for the first time. So when you jump in and ask your question, if you want to introduce yourself, that would be awesome too. And I would love to know where you are outside right now.
Still San Diego. It's nice here. I saw a few of you here in SoCal. I'm Travis Gerke. I'm also in the CRO space. And I'm a regular jokester here on the Hangout, in the chat usually. Yeah, this is awesome. I did have a question. We also moved away from 800-page PDFs to more HTML-based products. It's been a game changer for our safety monitoring boards. They really like it. But we're not as fancy as you. We're doing more reactable kind of static HTML documents, partly because of the challenge which you just discussed around validation of Shiny servers. So I'm glad Seth asked that question.
But one thing that's a gap for us, the feedback that we get from the monitors is still this kind of clunky, maybe they'll write a long email or give us Excel sheets or other things back. What are you thinking about or have you gotten any feedback from users writing your app, which seems like a good opportunity in the Shiny environment?
No, actually on the data points themselves. So a lot of times they'll look at a patient profile and go, ah, this lab value, are you sure this is right? Or, you know, just the usual kinds of, sometimes it's even data quality questions, which are valid because the monitors are often content matter experts, medical experts, and they see things that we as data scientists don't. It would be very convenient if while they're looking at your app, going through what was previously 900 pages of PDFs, if they could say, ooh, like highlight this and kind of submit right through the app and you could kind of get feedback in that format rather than more traditional methods.
Oh, I see. No, we haven't incorporated that in our app to accept feedback directly in the app from the DMC members. In fact, the feedback we've received are mostly not directly from DMC members, even in the email chain, because for pivotal studies for later phase studies, we hire an external CRO to interact with the DMC members directly. And we normally just get feedback from them because we're not invited to the meetings because we can't risk unblinding, even though the developers are not technically part of the trial team, but we are still part of Novartis business.
That's one of the things that's been part of my learning curve also, how many silos are intended in the process of clinical data review just to make sure data is secure. But I'm sure David might have something to say about building the function of feedback into the app interface to make it more user friendly.
Yeah, sorry, I probably missed the last part of Trevi's question about the user feedback, right? Not sure I followed correctly.
You want me to jump in, David? It was about when you all are giving these reports of, you know, like patient profiles, let's say, so what was previously long 900 page listings in PDF form. Now you're doing interactive, probably kind of more elegant web facing applications for viewing these. For us on the CRO side, which Beau helpfully just described as something you all don't have to deal with, we have to receive feedback from the monitor and, you know, safety committees. It'd be convenient if the Shiny app allowed them to do so directly in a way that links it to the data they were actually viewing at the time they gave feedback. Because we lose a lot of context sometimes when they try to tell us, you know, we looked at this patient profile and patient 001 has an abnormal lab value and we're kind of like going back through the data going, wait, what, you know, and the Shiny app seems like an opportune way to make that easier.
It is indeed. But I remember one project where it was for like specifically for team and then we had like a kind of surprising reaction, you know, from the end user, because they were actually not really convinced about the Shiny app. They would actually prefer using the PDF, the table listing, you know, these frightening things. Just because, yeah, there were like many options in the, like the safety explorer, widgets, stuff like that. Too many options, they didn't know where to click.
That's such a good point. We've gotten that feedback before. I was, um, it was making me think of a meetup we had before with Ryan at Veterans Affairs. I don't know how many hundreds of pages PDF into a Shiny app. That's such a good point. We've gotten that feedback before too. We built something really elegant. And then someone says, ah, but can't you just give me 900 pages and paper? I feel your pain.
I was, um, it was making me think of a meetup we had before with Ryan at veterans affairs on like going from this hundred multi, I don't know how many hundreds of pages PDF into a Shiny app to really like change the conversation with their executives, being able to bring that Shiny app to, to meetings. I just put the link in the chat if it's helpful for anybody.
Personal aspirations with R and Shiny
Um, but Marlene, I love a question you asked in the chat a bit earlier, and we'd love to have you, you jump in here.
Oh, sure. Um, David Bell. Uh, thanks for joining us. And, um, I'm Marlene Hankin. I'm over here on the West coast of Southern of the U S Southern California. Um, I usually ask the aspirational questions in the hangout. So I'm going to do that again here, especially with all the great conversation happening in the Shiny conference. I'm incredibly inspirational for people who feel like middle of the road, getting the hang of Shiny. Um, but I wanted to ask, you know, do you have any, um, like personal aspirations when it comes to using our, um, versus I'm sure you have corporate goals, right. Um, but like, what do you want to do on a personal level with, with your R and Shiny journey?
I want everyone to use good. So they won't send code to me and attachment. I don't blame you. Yeah. It's not specific to our, or any language, but I, I think it would make my life much easier.
How about you, David? Uh, yeah. If I think about, um, how's to do, um, Chinese have a pro migration. Um, I would say, yeah, uh, that would be good. You know, if everyone can use version control, uh, because sometimes I had to, to look for some code application, which was not even on geeks. Uh, so we exchanged the code by mail and I had to deploy the application on the opposite connect, which was, uh, you know, a little bit more challenging, which was, uh, if you imagine there are like 6,000 applications, um, yeah, it's, it's a bit challenging. Um, and then, um, yeah, it's not only geek, but also, um, package dependencies, uh, with RF, you know, uh, when you start your project on any project, not, not only a shiny ad, but it could be, um, I don't would be, uh, even a book for instance, when I wrote the book, uh, like I, I first thought about RF, you know, uh, freezing the packages I was
using because I knew at some point I would install a version which would break everything and it happens. And I was able to recover because, uh, because of that. Um, um, yeah, that can really make your life easier. Uh, that's also makes your life easier in CICB. Uh, when you use RF.
You can use a Revlog to restore packages seamlessly and then, um, works better with Posit Connect as well. So yeah, it's all benefits.
Thank you both for sharing. Thank you for all these great questions too.
Career advice
Um, something I wanted to ask, sometimes we ask this towards the very end, but I want to put it in here because Marlene, you reminded me, I love getting some of these more like human centric questions too. Um, but David, I'm curious, what is the best career advice you've ever received or advice that you've given?
And it doesn't have to be best. Maybe just the first that comes to mind. I was to remind myself not to use the word best there.
Uh, that's tough one. That's a tough one. Uh, that's not a technical, uh, advice I gave. Uh, that's just a mindset, uh, advice probably like never give up. Um, it's hard today. It will be better tomorrow. You know, these kind of things, uh, like Colleen said in the keynote, I was like, I always go back to the keynote, but, uh, there are days you want to quit, right? It's hard to make things work well, uh, using best practices. But in the long run, yeah, you learn about a lot about yourself. Um, you collaborate better with people.
Um, but yeah, I mean, for everything in life, it's hard. Uh, if you want to have like raise your kids, it's hard, uh, do sport. It's hard. So that's like a common point for everything. So believe in yourself and never give up.
So believe in yourself and never give up.
Thank you. Omar, I see you have your hand raised here. Do you want to jump in with your question?
Yeah. I wasn't sure what the proper core. No, no, that's great. And thank you. I can put the reminder here. Cause I know we're coming in at different times. Feel free to raise your hand on zoom. If you want to ask questions, you can also use the Slido link in the chat to ask anonymously, and then can also put them in zoom chat too. So thanks. Thanks Omar.
CICD deployment with renv and Posit Connect
Uh, so David, you're talking a little bit about the, uh, uh, RF and using, uh, connect for CICD sort of deployment. So the one thing with connect is it's. It's a poll. So it pulls from, you could, you could pull from, um, it's designed to pull things in. Uh, sometimes that's different from, uh, uh, enterprise sort of scenario where you have to, they're expecting everything to be packaged up and sent to the server. Uh, the, did you use orange to perhaps address some of those challenges around that? Um, just curious.
Yeah. So for instance, when we have, um, like your Shiny, for instance, we want to send to connect. Um, so first of all, we make sure it's using our end, right. We have frozen dependencies. Then we have the CICD pipeline. We have a specific section in the YAML file, which is restoring packages, uh, in the, in the runner. And then, um, you have some instructions to deploy the app, uh, on a Posit Connect. So you need, uh, like to use the connect API. Uh, so yeah, you, you need a API key, uh, on, uh, different things, not going to enter too much into details.
Posit Connect doesn't use our end. You know, when it restores the application, it uses backrats, but if you have frozen dependencies in the runner, it's much, much easier than if they are packages, you don't even know where they are coming from. And you try to really push everything. And then you have our messages everywhere in the Posit Connect. So yeah. From time to time, uh, you still have problems, right? But, uh, they are significantly reduced.
Great. Thank you so much for that.
Shiny best practices in pharma
I see a few hands raised, um, Eric, I know you had a question earlier in the chat too. You want to jump in? Yeah. I just saw that foyer of activity. I was afraid I was going to get lost, but, um, first, um, David didn't ask me to do this, but he did mention it. He has written this amazing book.
This, if you are doing anything intermediate or complex or shiny, you need this book, especially with someone like me who does not have any design background whatsoever or web development training before getting the shiny. It has been a huge, valuable resource for me. So thank you, David, for, for writing that, of course, um, to my actual question, um, sir, we've seen in talks either at the conference or in other venues that we're seeing shiny used in more places in the pharmaceutical, you know, area of development, as well, both saying with DMC interactions. So I guess in your opinion, what do you think are the biggest hurdles for those that maybe are coming new to pharma, but also new to shiny for learning kind of the best techniques to develop an application that's effective, especially with the nuances and situations that we encounter in pharma quite a bit. I'm just curious your take on that.
Um, it's more like the flow, uh, outside shiny. It's not just shiny, but, uh, it's a way you would learn shiny. Uh, when I learned at the beginning, I really learned reactivity. I learned that using reactive, using observe, observe event, isolate, but I didn't learn about RF. I learned later when I did the mistakes, putting everything to shiny, have the value to Posit Connect. I mean, my deployment fading because I didn't have the right version or whatever.
Um, this is kind of flow. Um, so when you, you see shiny, it's really, uh, if you ever did like PHP, uh, with my SQL, uh, you coupled everything together. It's not that easy. Uh, and when I discovered shiny in 2012, you know, at the end of the year, uh, it was super easy to develop this, uh, interactive application. You know, you, you do this, uh, you, you have the city input, he changed the table that just works in five minutes, but that's a trap, uh, you know, because yeah, in the long run, you would have more and more features. Uh, and then in the end you have state decode, which is harder and harder to maintain.
Um, so yeah, there is no strong emphasis around modules. Um, so if you, if you looked at Emily's talk, uh, from yesterday, um, yeah, modules should be one of the, like the first concept you should, uh, when you're done.
I see Eric clapping to that too. I've been a fan of modules as he knows. Yes.
Data team structure in pharma
So I know there, there's a lot of, uh, so many questions that we could probably make this hang out longer in the future, but I want to make sure I get to some of the anonymous questions too. Um, and I don't know if Bo or, uh, David, you want to answer this one? And then Mike, I see your hand, so I will definitely call on you next. But the question was, how are the data teams structured among engineering and analytics? Are they siloed or integrated in terms of the end to end project workflow?
Bo, do you want to answer that one or David wants to take that one? Repeat the question. So the question is how are the data teams structured among engineering and analytics at Novartis? Are they siloed or integrated in terms of the end to end project workflow?
I guess in a pharma, someone who's, um, also in the pharma industry, please correct me if I'm not, if I'm wrong. My understanding of a data, of a data engineer's role does not really exist normally in the pharma setting. Um, we have armies of trial statisticians who I guess falls more under the analytics side or, or bigger team. Actually our umbrella team is called, yeah, someone typed Cisco programmer. That's my assumption, but I didn't want to make that call in case.
Cause I, I've never worked in an organization with a team that's titled data engineers, but, um, the, if we're talking about people who create the data, curate the data, um, based on specifications, validate the data, um, and then, um, pipe them downstream for data scientists and, um, statisticians to create the analysis on. Yes. That team is mostly traditionally called SAS programmer in, in pharma. And they are two different organizations.
Um, there are different ways of organizing the two teams in different pharma companies. Sometimes they're in completely different, um, branches of the company with different set of higher management. So basically sometimes, um, the level of, of communication between stats programming and stats, uh, and statistics, um, which mirrors to data engineers and data scientists or data analysts in a general term. Sometimes it happens way up in the structure. Sometimes kind of in the middle, they're merged together into a bigger organization, but on a day-to-day basis in every project, in every clinical project, you would have an assigned programmer. It was the engineer who creates the data, um, and the statistician, um, assigned to each study, even though they're like management and management structure, they're in different teams with reporting to different, um, managers, but the two functions always work together on a clinical project.
Thanks. Well, I see there's a lot of thoughts shared here in the chat too. I wondered if, um, except you wanted to jump into with your perspective. No, I'm just, it was a minor, minor Adam. There's all sorts of pharma companies at all kinds of levels that all kind of do different things. Um, I'm sure from Novartis to smaller companies is different.
Yeah. And Eric, I know you mentioned what you call it at Eli Lilly too. Do you want to jump in? Yeah, it's basically, even though I don't quite agree with the term, they call that the data sciences group, but not the typical data science that you and I, or many in this call would expect, but they're the ones handling the day to day of getting the data delivered correctly or queries are answered. And they have their personnel embedded into like the trial team or the study team. So that each trial gets one of their representatives along, of course, stats and programming and medical and all that. So they're, they're pretty seamless in the trial process. I definitely don't interact with them as much from like a capability perspective, but certainly from a day-to-day trial perspective, they're, they're just part of it.
Modular Shiny app development at Novartis
Well, thank you, Mike. I see you have your hand raised. Want to jump in? Hi. Yeah. I mean, this is, this is going to be a softball question for David and Bo, but I'm. Super impressed by what's happening at Novartis where you've built essentially the, the data science team have built these Lego bricks for modules for building shiny apps. Then other people who want to build shiny apps aren't starting from a blank sheet, but they're kind of reusing those Lego pieces. Do you want to talk us through that? Cause I, I think that's a novel approach.
So Mike, when you say about the legal pieces, are you talking about data pipeline presentation from. Yeah. We had some presentation at our, some years ago where, yeah, there are some modules to like develop visualization for clinical trials. And then you can combine with modules together to create your application. So you don't need to, you know, develop them yourself.
So the modules we, I don't know how many modules we have, but we probably have like 30, 42 modules. Which you can basically plug into your app. And then depending what you want to do, if you're going to do like safety Explorer. Yeah. Patient profile. Whatever. They're very still.
Yeah. And in that case, people need to dive in the code. And at some points that's inevitable, but. Yeah. For sure. That's. A big change for us. The main challenge is also like making rules. Modules. Following best practices in terms of. Like software development. Everything, you know,
We are. You know, we are not even 10 in our team. So that's, that's a lot on our shoulders. Yeah. As Colleen said before, you know, From time to time you, you hear people. They're not super satisfied. Yeah. But in the meantime, you also have people that are really. Thankful to what you are doing. So that's. What's so. Rewarding for. As a developer.
We have a, have a bunch of. Of questions. I can kind of rapid fire, go through a few of them. Here, but. Well, another one on Slido was. My group is using BS for dash. And shiny pretty extensively. And I really love it.
The biggest frustration that they, this person runs into is. If they want to put in custom CSS. And David, they were wondering if you have any advice. On what they want to achieve. But yeah. If they want to have custom CSS in BS for dash. They won't be able to use BS. Because BS. BS for dash actually uses like. Refactored version of bootstrap. Specifically bootstrap. So if you ever use bootstrap. You won't have the same variables. The same survival. So it won't work.
So if you want to develop something for the long run. I would say you may want to go. Changing SAS. This aspect. So when I say SAS. It's not SAS. As a statistician. It's. It's SAS. Syntactically or some. Right. Another. Language.
It's a matter of learning curve. But in the long run. You better organize your style. In the app. And that's. The best practice to grow. If it's just like. 10 lines of CSS. You may not want to use SAS. So it really depends. How many. They want to customize.
Audience Q&A: getting user feedback
I love the. There's another question. I think it was from Damien. If you're still on. On interacting. Yeah, sure. Hey. Hi, everyone. My name is Damien. I'm the president of Epsilon. I'm really happy to be here. And really enjoying the conference so far. Right now. I'm in Bulgaria. But I'm actually on location. So I just came back from India and Vietnam. Vietnam. So really enjoying the remote setup.
I wanted to ask you about the users. Because the beauty of shiny is that we can build apps. Like very fast in small iterations. And we can keep getting the feedback and upgrading on them. And this makes it like a bleeding edge. Kind of software that we can provide to the users. And I know that you're also sure that it is not that easy to get. Feedback from the users directly. So I wanted to hear a little bit more about it. How do you cope with this?
So, yeah. Just to know whether your application are properly used. Or whether, you know. These kind of things. Right. And how people are using the applications. Typically.
So. This is. I'll be honest. This is something we will start to do at Novartis. So I did it. Like. In the open source. But I'm really talking. What we are doing at Novartis. So what we have. We have the tracker. In Posit connect. Which basically. It's an application. That we developed last year. And when I was in Japan. So, you know, when you travel, you can have. You can do many things. And yeah, this basically queries. The API to get. Like all applications. For all application you get. Who is accessing the application. How much time. Which basically give you some insights.
When you develop an application. Several teams. You can have some insight of. What which team is using the most. This application. And how you could target your support. Like if you see one team is using the application 20 times more than. Another one. Maybe. Like invest. More people to provide. Here.
You can apply to all applications that are deployed. That's not something you need to add to the application. And that that's important. That's something that works automatically. As soon as the app is on the server.
And on the other side. If you want to know how people are using the app. So like the in app usage. You may know. Like shiny head map. That's started from an experiment. I made around. The JS. Library. Because in the past, I was using. To track. On my website. And one day. I. Yeah, I was just wondering. Why. Why should. I just use. Sorry. I will choose. On my. So I tried the tracking. To put it in the. And then I couldn't see anything. It didn't work. And I connected the support. And they told me. Yes. It won't work. Because that's. Quite specific technology. And then I took. And brought some. Very simple code. We are in. Put it in a package. Yeah. The shiny head map.
So basically each time you click in your app. It will record. X and Y location. And then aggregate the data. At the end of the day. So you can see where people are clicking. Yeah. For instance, you can identify some dead stones in the application. Which will help you to. Refactor the design. At some point, you know, if you. If you realize there is a path that nobody visits. Maybe that's. Because it's badly designed. Because you know. For whatever reason. Yeah.
For instance, you can identify some dead stones in the application. Which will help you to. Refactor the design. At some point, you know, if you. If you realize there is a path that nobody visits. Maybe that's. Because it's badly designed.
So. Basically. What we would like to bring. And there is also user feedback. Like. Sending communications to users. So this one is a bit more active. You can be frustrated because sometimes, you know, you don't get the answer. People are very busy. And then. Yeah.
Wrapping up and final questions
We are quickly coming to the end of our hour here, unfortunately. So. Our positive team is talking behind the scenes that there's so many. Awesome questions here that we will think about how we can gather any questions that didn't get answered and maybe. Think about a blog post or some other resource. So I just wanted to let everybody know that in case your, your questions didn't get answered. There's so many great ones.
Niels. I see you had asked something in the chat. I think around. Using API's in addition to shiny. Do you want to jump in and ask that? Yeah, sure. Yeah. So I was wondering. Especially for bigger applications. If you want to separate surfer. From client-side rendering. Maybe add using. React as a front end and then separate it from the backend. You see. It's might be easier. Is that something you would do as well? Or are you only purely you shine?
From clinical from apps, supporting clinical projects. It's an easier answer is that we. Don't use anything beside shiny right now, because. First, we don't really have. Super big app, super complicated app and super big. Sample size. Clinical trial data are quite small compared to many other industries. And second is we actually tried react before separating. The front end and backend, just to improve. Flexibility and the visual effect and make it look better. But eventually, like I said, in the comments, our team is very small. Like David mentioned, we're not. Our goal is not to write a tool and maintain a tool for each study that comes to us, but to create something that we can ship off to our. Statisticians to use on their own and to manage on their own.
And we have much, much better. Adoption of our, especially with the younger version of a younger generation of statisticians coming in from academia. Many of them are quite. Fluent in our. But I think we haven't. Made the pivot in. Well, I don't think it's necessary for trial statisticians. Our is still. One of the best languages that they are the most familiar with coming. In from academia.
For some other bigger scale apps, like the one that David worked on for the HR data or the POS, maybe. You're right, boo. I totally agree. We decided to go for our. And if tomorrow we say people, yeah, we have to go to Julia or whatever. At some point, you know, there'll be a big class. Our is doing what we want to do. It's doing it well. Now we have many tools. If we want to speed up. We use parallel packages to speed up computations.
If we start to develop something with react. The problem is we are like three in the team to know react. So if we leave in the future, then who is going to maintain. The tool. So if we develop it in our, maybe it's not as fast. Maybe it's not as optimized, but at least we can find someone to maintain it in the future. And this is what matters, you know, when you're in the industry at some point.
So if we develop it in our, maybe it's not as fast. Maybe it's not as optimized, but at least we can find someone to maintain it in the future. And this is what matters, you know, when you're in the industry at some point.
Favorite resources for learning Shiny
Okay. Maybe a quick. Like 10 second answer from you, David and Bo. For, for teams who are just starting to learn how to use shiny. Or maybe helping somebody else learn. What is one of your favorite resources? David, you want to go first? Sorry, I had a cut. I couldn't hear the end. Okay, Bo, do you want to go first? What is one of your favorite resources for. Learning shiny or something you'd recommend.
For me personally. Stack overflow. But for someone who's just starting off, I think a lot of the, the, the gallery and the, and the tutorials on posits website. Are there. As I understand, like they're current, there are news articles. But you can always go back, go back to the. I think there's a series of what. 15, 20 videos. Of shiny tutorials from the very beginning. And I found that to be really helpful. Yeah, and later. At least mastering shiny, I believe it's in. It's on version two right now. A couple of years ago.
Thank you all for sharing the links in the chat too. As we go. That's so helpful. And I know Maya, you just mentioned our for DS community to help answer questions too. And that's an amazing suggestion as well. If we want to share the link for how people can. Join that community too. That would be really cool.
Awesome. Well, I know we are coming to the end of the time here and I don't want anybody to be late for the next. And I did just want to remind people that you'll go back to the shiny conference page and. Use hop in to get to the next event, but if you had fun today and want to join us again. We do have these data science hangouts every Thursday. From 12 to 1 Eastern time, so the same time and the same place. And so I'm just going to put this link. Into the chat if you wanted to add it to your calendar, you can do so there. There's also a data science hangout site on the positive. Site that you can check out recordings of past sessions as well.
With everybody here, I just have to also put a little plug here that the call for. Speakers is open for the positive conference in September in Chicago as well. So, just wanted to share that blog post announcing that too. Thank you so much. David and Bo for joining us and answering everyone's questions. Thank you all for jumping over here to the hangout. It's been fun spending time with you and. Seeing all the great discussions in the chat, not able to follow along with everything as we go. So I'm looking forward to reading it after as well. Thank you everybody.