image: thumbnail.jpg
Scaling a shiny app to 100K monthly users | Tan Ho | Data Science Hangout
We were recently joined by Tan Ho, Engineering manager at Teamworks Intelligence Soccer, to chat about scaling Shiny applications, working out loud and building a personal brand, defining "code in production," and best practices for R package development.
In this Hangout, we explore how Tan built a successful Shiny application that scaled to serve thousands of users, challenging the misconception that R does not scale. Tan shared his firsthand experience running the Dynasty Process Shiny app which peaked at 200,000 monthly users. He explained that the app was able to handle this load because user sessions were kept very short (around 30 seconds to a minute). Key advice for performance optimization includes focusing on small round-trip times for users, moving as much logic out of the app as possible (such as pre-computing or caching common requests), reducing the interface, and optimizing the actual queries (for instance, by rewriting them in data.table).
Resources mentioned in the video and zoom chat:
🔗 DynastyProcess GitHub → https://github.com/DynastyProcess
🔗 Tan Ho's Github → https://github.com/tanho63
🔗 Tan Ho talk: A minimum viable Shiny infrastructure for serving 95,000 monthly users → https://www.youtube.com/watch?v=zb3umo3Ka2g&t=3s
🔗 Tan Ho talk: Shiny in Production 2023: Tan Ho - Effective logging for Shiny → https://www.youtube.com/watch?v=wbUXrK_YWfI
🔗 Show Your Work by Austin Kleon → https://austinkleon.com/show-your-work/
If you didn’t join live, one great discussion you missed from the zoom chat was about display setups, with participants debating whether one screen is better for focus or if multiple screens (up to four!) are essential for data science work. How many monitors do you roll with? Libby can't live without at least two external monitors, but admits that sometimes just one laptop monitor is better for focusing.
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Timestamps:
00:00 Introduction
04:15 "Tell us just, like, a snippet of your journey into R/Shiny"
07:36 "How many users did you have on your shiny app?"
09:44 "How did the techniques and skills you learned in NFLverse help translate to your professional role?"
13:03 "Do you feel like working out loud played a big part in building a personal brand?"
15:29 "What is your top advice for thinking about code in production?"
21:30 "Do you have resources for people starting their Shiny journey?"
25:01 "What's the trigger point when you decide to bundle products into one package?"
27:25 "What were the biggest hurdles faced in scaling the Shiny app with high user load?"
31:23 "How can I do a better job of evaluating fantasy football players than ESPN projections?"
32:59 "What packages are most helpful for benchmarking and writing fast code?"
35:29 "Do you start with use case first or generic code and then refine?"
37:48 "What soft skills or personality traits help manage bugs efficiently?"
41:49 "Did you have instances when you felt you were going overboard with curiosity?"
45:25 "What are the best practices for reproducible production workflows for non-technical teams?"
49:00 "What piece of career advice would you give?"
Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12pm US Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
Can't wait to see you there. I am so excited to introduce today our featured leader, Tan Ho, Engineering Manager at Teamworks Intelligent Soccer, and you might also know Tan as a person who volunteers frequently at DSLC, Data Science Learning Community, and also is a huge open source contributor. So Tan, I would love it if you could introduce yourself, tell us a little bit about you and something you like to do for fun.
Yeah. Hey, everybody. Wow, there's a lot of people in here. My name is Tan. I am based in Ottawa, Canada. I've been writing R since 2019. And my background is in business. So I studied commerce as an undergrad. Then I went into my family business, and then the family business was property management, and basically hated that job. So along the way, I started doing, you know, hobby stuff in fantasy football, and, you know, building spreadsheets, doing sports analytics. And eventually that kind of snowballed as I got more and more skilled at spreadsheets, you know, the demand for, you know, doing some programming to support it kind of increased. And eventually that led me to starting work on, you know, programming and fantasy football packages and so on.
And then along the way, basically, I lost my job in property management, and then proceeded to find a new job as a construction data analyst. And that has since snowballed into a position with Zealous, which is now called, which was acquired by a company called Teamworks. And now I work professionally in software analytics. So that's the quick rundown. And you asked about something fun I like to do. Definitely, this time of year is prime pumpkin carving season. And I am starting to kind of figure out what pumpkins I want to carve this year. Generally, I try to carve at least one, I think this year, I want to try for two. And I haven't really figured out what those things are going to be yet. But a couple ideas. So we'll definitely take some suggestions if you have any.
Yeah. And also, if you happen to have carved pumpkin pictures right now that you would like to put in the chat for my enjoyment, I would love them. I absolutely love pumpkin carving. I put a link to Tans pumpkin carving website, like I'm going to put it in there just right now again, in case anybody missed it that first time. Go take a look at Tans pumpkins. They are amazing. Even though I have a very bad track record with recognizing what they are, when they're tiny on the screen.
From Excel to Shiny-R: Tan's origin story
Okay, let's dig in just a little bit more with that background because I want everybody to understand just how amazing it is that you started somewhere that you hated doing something that you didn't like to do a previous life, shall we say to data science, and you built something in Excel with like, power query and stuff, right? Not even an R, you did not know R, you did not know Shiny, you built it and you got so much of a following on Reddit and so many people using your thing that you're like, we need to scale this up, right? How amazing is it that that's how you learned R? Tell us a snippet of that.
Yeah, it's really crazy because like, I didn't have any programming by trade, right? So I went to business school, I was good at Excel, but you know, I wasn't really sure what I want to do with myself aside from that. And so I started, you know, essentially like trying to connect to APIs through power query. So that's the power BI, like querying, like data manipulation language. And that was kind of my gateway drug into like some of the programming stuff. Because, you know, at this point, I had like, I want to say like over 20 fantasy leagues and to try to analyze all of them basically just needed scale, right? And so as soon as you get one league, you can just, you know, do it manually. But when you get to 20, you're like, okay, now, now you're a programmer. And so I put that all together, put it on Reddit, and people started asking me like, hey, can you make me a sheet just like this? And it works fine the first one, two times, but power query at that time, was basically just on Excel for Windows. And so it was kind of challenging because, you know, there wasn't a Mac version of this or wasn't, you know, they had one mobile or whatever, right?
And so built enough of a following. And so one of my friends, Joe Sulewski, who I also happened to meet on Reddit through a fantasy league, basically just offered to write me a Shiny app. He was learning R in school. And so he learned R and a little bit of Shiny at the time. And he wrote the very first version of the Shiny app. And so at this point, he basically built a front end that hooked into my backend app, like my backend, like spreadsheets and data polls and so on, and put it on the internet. And it basically exploded. And he was a grad student at the time. I was, you know, working on, I was working on, you know, as a product manager, just doing stuff as a hobby, but still doing this, mostly with Excel. And so he couldn't keep up with essentially feature requests. And I was like, well, dude, like, this thing's broken, you know, you need to do this, it needs to do that. And I just became that really pesky project manager. And so one day he just gave up and was like, just go do it yourself. Right? And it's like, okay, sure. Where do I do? What do I do? And so my very first experience with R was basically like bug fixing and editing the Shiny app and didn't use tidyverse. It didn't use, you know, it just used Shiny. And it was like this awful base R thing. And he'll forgive me for saying this, because we're both, you know, whatever. But I basically like taught myself with this app with it, I needed to do this and taught myself Shiny first. And then I taught myself tidyverse. And then basically rewrote the whole app eventually to use tidyverse and like to build like this data pipeline that went with it. And that was our first big app. And so my gateway drug to R was a Shiny app that did fantasy football. And then after that, I started expanding from there.
Scaling to 200,000 monthly users
Wait, wait, wait. Important point. How many users did you have? Because I have worked on apps professionally that I've had like seven users if I'm lucky.
Yeah. I mean, I think it hit, it got called to death basically immediately when we put it in. And that's like Joe's first version. How many thousands of users, Tan?
And so at its peak, I think it hit 200,000 monthly users on the Shiny app itself, which was spread over, I think, three app instances at that time. And so that's like with Google Analytics monthly users. But when people talk about Shiny not scaling, I laugh, because I have firsthand experience running an app out of my garage, essentially, uh, with hundreds of thousands of users and like pictures to prove it and everything. And to this day, it's like a hundred thousand users. Yeah.
But when people talk about Shiny not scaling, I laugh, because I have firsthand experience running an app out of my garage, essentially, uh, with hundreds of thousands of users and like pictures to prove it and everything.
There's a lot of like faces in our grid of people that was like, Oh my gosh, that's amazing. And I got a little, you know, seven users. Libby, I've been there. Yeah. I mean, I think that getting people to use something that you built at a business that's like a dashboard or something is really, really hard. I'm going to put a link in the chat here. Everybody go save this talk for later so you can watch it, um, and get a more in-depth view of what Tan's talking about this Shiny app that was serving a hundred thousand monthly users. Um, okay. Eric, you had a question, Eric Nance, who is another, uh, open source, our hero. Would you like to ask that live?
Sure. And, um, for those that may not know Tan and I go pretty far back and he is larger responsible for me getting that live streaming bug. So it's, it's all your fault. No, all good. But my, my question is obviously I've been a huge fan of the open source work you've done, especially around sports analytics, your contributions to the NFL verse, even though I'm not a football fan, primary, I love the tooling around it. Just care, especially for those that may be wondering, just, you know, this out loud, so to speak, how did the techniques and maybe skills you learn in crafting a lot of the tooling behind NFL verse help translate to when you actually were able to get a new role that, as you mentioned, developing these sophisticated R tools or, or, or programs, what kind of techniques and tools and, you know, ideas that you learned from your open source work that have directly benefited in your professional day today?
Lessons from open source applied to professional work
Yeah, it's interesting, right? Because like nothing I can do. So I work most on internal tools. And I think a lot of people will have, will share this experience, um, as our production engineers essentially is that your user base, especially with Shiny apps, but even just pipelines and stuff is mostly internal. But I think all of the things that I learned working on dynasty process I use today, like every day, right? Like all of the stuff, learning to run your own server, how do you scale things? How do you, um, like obviously like performance, like I write fat, like I write fast code because I had to, right? Like I had to serve, I have to write code that serves that many people. But I think the other thing that it really taught me was just the interface part, like getting people to adopt your use. Like, I think the parts about the packages are very interesting because I I'm on a team now of essentially 10. Our coders essentially. And I think all of the stuff that NFL verse and open source packages have taught me is to really think about the way you design that interface, trying to allow them to basically go through and do whatever it is they need to do. It's part of the, like building the tooling and build thinking, coming at it as like a tools process is part of that, right? Cause I think the temptation from like an, like if you only write analysis for yourself is, you know, you write it.
Basically like you write what you need right then. And I almost by default now we'll write, okay, well, what's the like building blocks that makes sense for this general process. And so coming at it a little more procedurally, a little more like what's the interface going to be, whether that's an app, whether that's a package, whether that's like, you know, what, what are the user stories? What are the expectations? That's sort of the main takeaway that I think across the board, obviously like the process of design pack is also, you know, the branding to help me get the job and all that kind of stuff. But I think as a whole that would be like, yeah, I would probably point back to the idea. The main idea is basically like, you know, thinking through user interface story, how they're going to work with your work.
Working out loud and building a personal brand
I think also it's really important to point out just how much Tan worked out loud. So you started coding in R or learning R in 2019, I think. And then during the pandemic, you started Twitch streaming your coding, which means that you were working out loud very early, like probably before you felt like you were a confident coder. Do you feel like that had a big part to play in your confidence level in your, you know, building a personal brand being recognized? Definitely. And I think like the streaming has paid off in a whole bunch of ways, but I think the, the number one way is just like this ability to kind of talk through when you get stuck on things or, you know, pair program through like when you, when you run into a bug, you know, there's no panic, especially like if something's broken, you know, it's often quickest to just think through what the problem is and solve that thing. And I've definitely found it useful. Like the streaming for sure was a big part of that. And I also do it, you know, asynchronously in, you know, forum questions and try to think through other people's questions and, you know, help build those mental models and so on. The branding part for sure. I think part of that, like, I think the, the, the part with a hundred thousand users and whatever, I think like that's a different set of people, right? Like those are people who are mostly interested in fantasy football, but definitely the packages work, working out loud, building public tools. That was a lot of what I would say, like built towards the like public coding reputation and, you know, trying to be approachable, trying to pay all of that forward is one of those like big motivators that I think has just had a ton of benefits every time you turn around. Sure. Yeah. So do it, work out loud, everybody. Yeah, absolutely. Be brave. We're such a fun and supportive community that I really encourage you to do it. And we're going to have a friend of Tans on very soon. The beginning of November, Arielle Drawer, who also very heavily worked out loud and within like six months of
working out loud, had a job herself working in soccer analytics. So lots of success stories for working out loud. And if you would like to get a start, share something on Blue Sky. Blue Sky is a really great place in the RStats and data BS communities.
Well, I wanted to bring in some questions from Slido. Don't forget, you can ask questions on Slido, everybody. And we had an anonymous question that was, what is your top advice for data scientists who want to get better about thinking about code in production specifically? Or do you have any resources that you use to help you learn about code in production?
Defining "code in production"
That's a great question. I think like the part about coding and production, I think it's hard for like data scientists sometimes because it's hard to visualize what production is, right? And so the way I like to think about production is, you know, you have an analysis, it becomes something in production when someone uses it to make a decision. And so if you put that analysis into someone's hands and they're looking at it and doing something with that, based on what they're reading from you, that becomes production.
If you're just writing, you know, if you just put a plot for it being pretty or for, you know, communicating something, the moment someone reads it and thinks that they need to go do something, that is the start of the production pipeline, if you will. And so then it's like, okay, well, they made that decision once, but they will probably want to do that repeatedly. So then, you know, thinking about the reproducibility and, you know, pipeline and consistency. But I think in general, everything that influences somebody to make a decision from your analysis is production.
And so moving that, I want to call it a goalpost. I'm going to have to make a lot of sport references, I'm sorry, in advance. But moving the goalpost from this mythical, you know, like code in production is an app, it's a report, it's a, no. Code in, like, the data science part that when it is in production is when someone reads it and makes a decision from that. And so hold yourself to standards for production level thing if they're going to do something with your code.
If you're just writing it for fun, if it's, you know, a, my very first package, pseudo package essentially was like a dad joke API app or whatever, right? That's not production. Then I was going to make a decision from that. That's, that would be something I would consider a toy. But if someone was going to do something with my data viz, with my report, with my analysis, that's production. And so anytime you get into something that's going to do that, don't take, like, that's when you need to like hold yourself to whatever your standard of production is.
And then, you know, that those standards will grow, the competencies will grow as you work through that process. So like my definition of what production quality is might be different than yours, might be different than any other person, but when to apply that standard is sort of my like thing that I think people should do more of, which is like, anytime someone does a decision, you should do that. So that's, that's when you need to put it in a package. That's when you need to write functions. That's when you need to document. That's when you need to leave it notes for yourself, right? Like all of those things. And once someone makes a decision with it, that's production. So hold yourself to production standards at that point.
I think in general, everything that influences somebody to make a decision from your analysis is production.
I feel like that's good advice for anybody thinking about anything. Because a lot of times, the thing that you start on that you're not thinking about is going to be in production might end up being in production. And I've worked for a lot of companies where they have tools that are sort of like glued together with spit because 20 years ago, somebody built it on the side and now it is a production thing because like, oops, that was really helpful. Now we're just going to use it. And it wasn't really ready to be a production tool yet.
Um, so if like, survives, if you label it a prototype that you've just cursed it to being used 20 years ago, basically. Yeah. Rachel even gives me this advice when I set events up in zoom, she's like, don't set them up as a pretend one, set it up as the thing because it's probably going to become the thing, which is a really, really good piece of advice.
Working out loud and show your work
Well, I also wanted to share in the chat, um, the book show your work by Austin Cleon, because tan, this is something that you mentioned to me and you're not the only one. I think that, um, car Thompson also mentioned this book to me. And it's one that's been on my to read list for a while. So if anybody else has read it, let me know.
Oh my gosh, Tim just shared so many pumpkins in the chat. Everybody go check out pumpkins in the chat. Um, I love show your work. I have it like right next to me, like all the time. And it is literally the world's smallest, like book. If you can see it here, yeah. It's very cute. The font size is huge and has pictures. So totally like super, it's like super quick read, but I think it's really good for like the marketing and like really what I would consider like authentic self-promotion marketing is that like, you want to demonstrate, you know, I'm a person, I think through things I am, you know, you make yourself more relatable by working through things in production, like working out loud, sorry, got production on my brain.
And, you know, like it's, it's super good inspiration for just go out and do more stuff, right? Like I wouldn't be where I am without, um, having worked in public on this on Twitter, on Slack, on whatever. Right. So most of those things are all, um, you know, because I work out loud. Sure. Yeah. I mean, same. Um, all right.
Shiny resources and learning
We have a question from newer in the slide. Oh, nor would you like to unmute and ask your question live? And I want you nor to direct this to everybody as well. I think we could all share some resources here and Eric had already shared one in a reply as well. Thank you, Eric. I'll put that in the chat too.
Hello, Tan. Uh, so before I crack out, uh, I, I know of one book mastering shiny, but do you have any other good references or resources for people who are starting their shiny journey? Or is it just sort of like build something, break something, learn from this, maybe do the next one better.
Yeah. And this is for everyone. So, yeah, I definitely those resources didn't exist. Mastering shiny is an awesome resource. Um, I think I'm a person who learns by projects and helping people like answer questions. Those are kind of the two main things that I like think through. I'm very like, so very like case study situational type stuff. I'm bad at textbooks. I am, um, not very good at those kinds of things, but, um, I know there are a bunch of good resources. I think, you know, mastery shiny is probably the book that gets you from zero to 80 these days.
Um, and then from there, you basically start getting more advanced at things. There's a great book by David Grand John, who's basically God of our, uh, of shiny UX and UI, um, call, I think it's outstanding user interfaces. Um, and I have read that I peruse it every now and then. I think that's like a super useful way to think about those things.
Um, the backend stuff, I think like, actually, you know, there's probably my, my thing about the shiny backend is that actually people are overcomplicating it in most cases. Um, I think that your backend should be less complicated, move that stuff into a package is my like main thesis there. So I would encourage people to go into our packages next, actually, um, in terms of like production, like, you know, putting shiny apps in prod.
Um, those are kind of the two places that I go to most. I think, you know, I advise working on projects, learning something, um, figuring out what you need at that moment. And then also just like going around and looking at other apps and other websites, right? Like, I think like if you put yourself into that user brain, I think that like, that's, that'll teach you a lot of like, you know, design type stuff. Like people ask, we'll sometimes ask like, how do you be a good like app designer? I'm a terrible one, but like, I think like the ideas are like, there are a lot of people working on really good stuff. How do you emulate this point click information, that kind of thing. So those are kind of the main things I think about for sure.
Yeah. And I think that, um, volunteering can also help. Like if you have a buddy who's working on a shiny app volunteer to work on it with them, you might learn things. Um, like I didn't learn modularization in shiny apps until I was working on one for the first time. And I was like, I guess I'm going to learn modules now because I have to do them. I think that, um, the working out loud part can be with buddies. It doesn't have to be alone. You can, you can do it with friends. I think Ariel is going to be a good example of that on, on her hangout as well, where the community is like a feedback mechanism, right? Like you do something, you get feedback about it, you get to iterate on it.
When to make it a package
And you mentioned going into packages next as a sort of bit of advice. And I know that Travis had asked in the chat, like, well, what's the trigger point where you decide to bundle several packages or products into one thing? Like NFL versus now, a lot of things kind of like tidy versus a lot of things. Um, how big did it get before you decided to put everything in one?
I start with everything as a package. Now I find that like, it's a little bit like everyone is, I think this, this will sometimes seem like dogma where I think like, you'll see people tell you like, just start everything as a package. Um, but there's a couple compelling reasons for me that like everything is a package is just the right way to think about things. Um, one of them is that you can like, there's like a bunch of resourcing built around packages, right? Packages, a package is a collection of functions to me. And so as soon as you have more than one function, you have a package in my opinion. And so you can just put that right away.
Um, the, the definition of package being just, it's a collection of functions also means that like, that's when you can start thinking about how do I test this? How do I document this as a function? How does someone use this code? Um, people think about like, it needs to meet all of the CRAN bar of like standards, um, which I don't agree with. I think like a package can be just, you know, a collection of functions with documentation and maybe tests and that's about it. But like, when should it be a package from the start is really my answer. And I'm very much on that, you know, a bunch of people. So.
I've noticed that as a trend in my friends who are shiny developers over the years have moved further and further towards every shiny app is a package and everything I do is now a package. And anytime I write something that's more than one function, it's a package because your documentation is bundled in with it. And it forces you to write the documentation for the stuff, the descriptions and everything. Um, and I think that that's a really, really great benefit of going the package route.
Scaling Shiny to 100K users
Um, okay. We had a question from Hubert in Slido. Hubert, would you like to ask your question live?
Hubert Hickman. Yep, I will. Um, so my question is where you said that user, the number of users you have and immediately got a headache just thinking about it, but, um, what were the biggest hurdles you faced in getting the app to run successfully with that user load and how much did the characteristics of that user base drive this? Is your user base, is your user base, was it primarily, are they in and out or do they stay in the app using it a long time? How did that shape the, uh, how you tuned and made the thing work?
Yeah. And so the most popular app is essentially a trade calculator. And so, um, as a brief explanation of fantasy football, you basically, you know, you get players on your team and you get points from them doing things on the field. And so each person in a league will have players. And, um, a lot of what the, well, the very, the most successful app is basically just, I want to trade this player for that player and this other player, is this fair?
And so the individual requests are, was quite quick, right? Like some of the math stuff behind it, you know, I could pre-calculate and all of that stuff. And so the initial, like each user session is quite short and, you know, their interactions make for very small request turnaround times, but some like some thoughts around that essentially are like moving as much of the logic out of the app as possible. If you can pre-compute it, great. If you can, you know, um, cache common requests, that's great. Um, if you can, um, you know, simplify the interface or reduce the interface that serves the thing so that like they only really get to play with like a limited set of things, then like you do a little less, but you do that thing really well.
And then the other part is like, if you can speed up the actual query, so we wrote it in data table, for instance, um, that scales quickly. And so Shiny is really good at, like, it's easy to, it's, it's the reputation about scaling definitely is partially like how long is the session. So the average session is like 30 seconds or a minute or whatever. They get the answer and then move on kind of thing. Um, but like once, if the app is not doing any thinking, it can serve a lot of users really well, right? Like it's, it's not when it's, when everybody's idle or roughly idle, it's not that huge a pain to have everybody opening the app at once. It's how many simultaneous calculations can
you have at once? And so then, you know, that's where you get into promises or, you know, horizontal scaling and like, you know, pushing everybody to different instances of that app. Um, but as a general thing, like, yeah, it's the reason why it's so successful is because it's small and does a very short calculation that returns a thing. Um, and that's like the main, like performance consideration is like making each user's like round trip very, very small.
Um, I love it. I, this is also something that you mentioned in your Shiny in production talk, which is like the, don't have it do a bunch of stuff. You're making Shiny do too much. You could have that all happening somewhere else and Shiny could just be doing the stuff that's serving the people, the thing. I think that's fantastic advice. Um, and I think we had a, a specific fantasy football question. Who was it? Joe, Joe Harder, you would ask us, would you like to hop in and ask that live?
Yeah, sure. Um, I don't trust myself. So when ESPN says projected points are whatever for this coming week, I just say, okay, I just go with the high, the top one. But have you found that you can, you can do better and like, how can maybe I, I do a better job of evaluating players?
Yeah, I think that like the humility part of it is actually very accurate. Right. So I think if you think that you're not smarter than the crowd, you tend to do better than if you, you know, go off on your own, which is actually kind of a counterintuitive thing to think. Um, so I tend to pick lineups and stuff like that based on like consensus rankings, fantasy pros, ADP, that kind of thing. Um, and do a lot of my own modeling just based on like rankings actually more so than like individual features. I think it's, there's a lot of noise, um, in that space. And so if you can lean on wisdom of crowds, I think that that's, um, generally going to end up with a slightly better output than like ESPN, which you don't really know, like what is going into that. Right. So some computer thing that might be waiting, whatever. Um, but to me, I think like wisdom of crowds is most of where I would go for fantasy stuff.
if you can lean on wisdom of crowds, I think that that's, um, generally going to end up with a slightly better output than like ESPN, which you don't really know, like what is going into that.
All right. Awesome. Joe, thank you for asking. David, you had a question, David Geerarts. Would you like to ask that live? If I can unmute. Yep. Okay. Hey, I got excited when you talked about writing fast code. So I'm curious about your experience with any particular packages that you found most helpful. And if you, uh, if you have any insights into pros and cons with respect to like just time, the whole project versus timing specifics, the bits of code.
Yeah. Any insights you have? Yeah. In general, when you're interested in code timings, you either approach it from inside, like, is it like a single function or you approach it from like the code performance in the bigger system. And so, um, the tech, the tech techniques I use the most is like bench, the bench package to benchmark something to specifically like that set of things. Um, and that's good too, because it also gives you a nice, um, like sort of estimate of like memory usage. Um, and it like, we'll like run multiple iterations of the code rather than just once. So the bench package is my go-to for like timing single function. Um, and then actually like, I just use log messages most of the time with time stuff for like bigger systems.
Um, I think that like, it's hard to get a sense of where something is slow. So, um, if you stick a bunch of log messages in, I'm excited about the new open telemetry stuff that Barrett is working on. I'm sorry for butchering his last name there. Um, but you know, that's a similar thing where you're basically logging like code block timing. Um, those are kind of the two things that I'm most, I use the most. Um, but in general, like I don't really optimize something until I see that I need to. And so that's where like, I start with logging and, um, would be keen to try open telemetry for sure for that kind of stuff. Uh, and then once I get into that, I can sort of identify where I should get, where I should spend time to make things go faster.
Awesome. The, the making things go faster thing is my downfall. I tend to write stuff that just works and then hope that somebody else is going to help me make it faster.
Thank you, David, for asking that question. Um, and there was one that I liked that was about the refining process. Um, it was an anonymous question and it said, do you start with use case first and then scale or generic code and then refine?
And I'm guessing that this is, this is about your initial development process.
Yeah. Generally use case first. I I'm the, I'm the kind of person who will, for shiny apps, especially I will wireframe them out on a tail draw or lucid chart or something like that first stigma. Um, now, um, and then hook things into it after, but even for packages, like, you know, I want to, I have a rough sense of like how I think people will use it. And then, um, you know, start with those building blocks first. Um, people, some people are, um, test driven development. That's sort of the same idea. You write the test first because the test is the use case, right? Like you can do that. I know my friend, John Harmon would say something about vignette driven development, which is the same idea. It's a similar idea about like, what's the use case, like draw the story and then build the functions that support that use case. Um, definitely that is how I would think about it because sometimes you can start at too generic, you can start at too high a level or too low a level. Um, so I would definitely start, uh, with use cases and user stories. Sure. Dan Chen says conference driven development.
Conference driven. Yes. Yeah. If you have to present it at a conference, that is very much a use case, like, you know, shoving it in front of end people explaining why code is always like the first step for sure. So sometimes that's why working out loud is helpful because you know that other people are going to look at it and you're, you're like, all right, well, I'm going to on purpose, give this to other people to look at it. And it kind of forces you to clean up some stuff, um, and organize some stuff a little bit better. Okay. Thank you so much. Anonymous ask her for that one direction. You had a question inside. Oh, would you like to ask that one?
Soft skills and self-teaching
It was a soft skills question. Hi, I hope everyone is doing great. Hi, everyone. Okay. So my question is as a new yourself, uh, as a self taught programmer. So I'm not going to directly ask about data and stuff. I'm going to ask about the soft skill part, like personality trait, like how right now you manage issues or bugs or break in pipelines, et cetera, as compared to when you first started, like, or any personality trait you adopted during this journey that make you more, uh, I can say efficient in handling these as compared to when you first started learning coding and stuff.
Yeah, that's an interesting question. It was almost like, how did you grow as a person through this Dan? Yeah, it's interesting. And I think like, that's, I think, um, I don't think I'm that different from a traits perspective. I think the things that have helped me the most, um, I, I mean, part of this is like ADHD-ness, but like this, like curiosity and wanting to understand the whole thing. Like I want to, like, I want to go deep on basically everything. And I think like, what that actually is, as I kind of learn more about programming is this idea that, um, mental models are everything. Um, and really fleshing those out takes a lot of persistence, curiosity, you know, willing to help, right? Like a huge part of like building out that skillset is just lurking the DSLC forums and, um, or, you know, help channels and trying to get through everything just to kind of test my own mental model, right? Like some of like part of the, like, you know, things around, like answering questions and, you know, being helpful and paying it forward. Some of it is like motivation of the motivated by helping people. Um, but I'll admit to being really selfish about some of these things is in that it helps me really like test depth of knowledge. And, you know, really, if I can like understand the problem and explain that mental model, um, it really kind of strengthens your own learnings in a really good way.
mental models are everything. Um, and really fleshing those out takes a lot of persistence, curiosity, you know, willing to help, right?
And so, you know, very curious, um, very, you know, driven to understand something to a lot of depth. Those are kind of like some of my, some of my, some of the things that I think are the most useful from a self-teaching perspective is just like go through, build mental models, understand the relationships between things, why things work the way they work. Um, and it's actually one of the things that like LLMs concerns me the most about is that like people are less likely to develop strong mental models if they, you know, part of their mental model becomes use LLM to understand why this thing is, then just ask LLM and then come back. Right. That's one of my actually biggest concerns about that and the loss of the critical thinking curiosity parts. Um, but, you know, patience, you know, I think that that's definitely a, you know, and like lack of shame, right? Like we've talked about the like learning in public and live streaming. Um, those things I think are also useful, um, not specifically about the self-taught like part perhaps, but, um, you know, I think that those skills and soft skills are great for more for an advancement side of things perhaps, or, um, you know, calmly approaching bugs. Sure. Um, but also having faith in yourself that you can solve basically everything. You solve a lot of hard bugs before, and you can solve anything that you try to.
I like that a lot. Have confidence in yourself. You have a track record of getting through a lot of bugs. You'll probably get through this one. Um, thank you so much for asking that question direction. I love it.
Going overboard and the 80% rule
And we have a few more questions that I bet we can get in, in our 12 minutes that we have left. And one of them is a follow on to this. So I'm going to ask it. It's an anonymous question that says you mentioned curiosity and going in depth. Did you have instances when you felt you were going overboard and had to pull back a little and as somebody who over engineers.
Uh, I can relate. I think that there's a, there's always like how curious, like, I think if you are, if you are still feeling curious about something, it is. Perfectly fine to keep going in my opinion. Um, it is, I, I try to draw lines around when I find it detrimental in the sense that, like, I really don't understand. I'm really stuck or whatever. Like, I think that. That sort of lines where it's like the frustration line is kind of one that I monitor a little bit where, like, if I'm too frustrated to solve this thing, I need to take a break. That's sort of like, you know, when I want to pull back or, you know, what is the, like trying to weigh it against like. Engineering optimal versus like, what is the 80% line? Um, so to the. This is also from John, um, introduced me to this talk, but, uh, this video, but, um, a project is like 1 golden rule for like this sort of thing is always ship a project when you think it's like 80% done and then take a break from that project. So like, there is. Like this thing that people always want to make it perfect to do this thing, but when it's 80% of as good as you think it could be based on where you are now and all your skills ship it. And then, you know, if it's really, really, really.
You know, something you want to come back to, you'll find the time. After you take that break and like, you'll come back to it and like users will drive you back to this thing, but always, always, always, always ship it, especially at the 80% mark, because that last 20% is going to take 80% more work essentially. Right. So like the 80, 20 rule of like shipping things is like always ship it at 80%. And then don't go back to it for a long time, essentially. Um, and that's been super helpful to also stop me from going overboard on how deep things to go.
that last 20% is going to take 80% more work essentially. Right. So like the 80, 20 rule of like shipping things is like always ship it at 80%.
Yeah. I like that. I, I tend to allow myself to get caught up in the pressure of needing to go fast. So I stopped myself from following my curiosity into the like rabbit holes that it wants to go into. And I feel like in the longterm overall, that makes things worse for me because then I'm like, okay, I'm going to look for a shortcut to solve my problem instead of learning these parts and pieces more deeply so that I understand them on a functional level, foundational level, and how they fit together. Um, it's like, if you go and you try to understand modules and shiny, and you're like, I'm just going to try to skim this, you're going to end up writing terrible, shiny code. And it's going to be really frustrating. And modules are really going to frustrate you. But if you took the time to really, really in-depth learn modules and understand both modules and shiny and whatever your subject is that you're building your app around, you're going to have a better time overall, not taking the shortcut.
Um, so I think that that is, that's my two cents on that one. Thank you for asking that question.
Anonymous asker. I appreciate it. Edward, I saw you had asked a question in Slido as well. Would you like to unmute and ask that live? Yeah. Hey, cool. Um, so I, my question was just, uh, after you built a prototype and you need to move something, uh, it's like more robust for production. What design ideas and tools are the most important for making sure your workflow is reproducible and debuggable, especially when you're working, you're handing off the project to teammates with no software engineering background. I work in public policy and I'm like the only one on my team that has a background in software engineering. And like the, to the point where like object oriented programming is like, uh, was an unheard thing of when I showed up. So this question is just like, what, what can I do to as best practices for the rest of the team?
That's a great question. Um, in general, I think that if you're handing a software, like, are you talking about handing an app off to a entirely non-technical team? Because I think that's a slightly different situation than handing it off to technical users who are not as technical as you, right? So I think that's actually two different things almost, because if you're handing it to a non-technical team, you want, um, basically you want to give them something like a Shiny app or like a public, like you only want to give them the quarter output or the markdown output and not the actual app, um, or not the app's code. Right. And so to me, that means you need to have it deployed somewhere and you should hand it off there and, you know, have a feedback form, a Google slide, a Google form link to collect feedback on the thing, um, and have a way for them to communicate thoughts around that prototype. Um, they're going to run into bugs. And so logging is more important in that case, because they're probably not going to be able to like reproduce it. Like, they're not going to hand you a reprex. They're going to be like, you know, I did this thing and it broke. Um, and so to me, like when you hand it off entirely non-technical, that's sort of my like lines of thought is that there's the polish required from you is a little higher and you're going to need to guard against, like, you're going to need to find ways to collect their feedback that are non-code oriented.
Um, as far as like less technical teammates who are still coding, um, I think that's where like setting production standards, um, become a little bit of a thing and, you know, being good at templating, being good at having tests and, um, you know, I dockerize everything. And so if you have those things structured and, you know, as many guardrails as you can set up, um, you know, that doesn't mean that they need to understand those things in order to like work with the code. But, you know, I think having more guardrails and, you know, supporting their development as technical coders becomes more important than, um, more so than, you know, the polished layers of like the app kind of thing. If you're going to hand a production script, right, like that's sort of more important to have those tests and things in place, um, and containerize and the deployment parts handled kind of thing rather than like, you know, a Google form or something like that. So.
Career advice: make friends, not contacts
Nice. Thank you. Thank you, Edward, for asking that question and thanks, Tan, for the answer. And now with our five minutes left, I would love to ask you the question we try to ask everybody, which is, do you have a piece of career advice that you either have followed and has done you well or that you like to give others? We talked about sharing your work. We talked about working in public. Um, you know, I think the one that, the one that's that we haven't already touched on, uh, yet, I'd say is making friends. I think make friends in your communities, um, make nerdy friends, talk to people, talk about the things you're interested in, talk with, you know, people doing the work that you're interested in, understand, like, sort of how they think, make friends. It doesn't have to be friends. You don't have to be friends with, you know, people who are like massively further ahead of you, but I think like making friends with, you know, your peers who are looking in this space being, you know, open and, you know, sharing your work and, you know, sharing memes and those connections matter more.
Then, and like, when I say making friends, people go like, you know, when I say making friends, I don't say networking for a reason, right? Like a few really good friends are going to be way more than, you know, a LinkedIn button acquaintance kind of thing. So to me, like make friends in the spaces you care about, they, you know, will tell you to, you know, go fix the dang app yourself and, you know, do those things for you that like, you can't force, I think, in like a network, in like a more, you know, over networking way. Like, I think if you think about networking as go make nerdy friends, having a few really good nerdy friends will take you so much further in that space than in any other way.
Like a few really good friends are going to be way more than, you know, a LinkedIn button acquaintance kind of thing. So to me, like make friends in the spaces you care about, they, you know, will tell you to, you know, go fix the dang app yourself and, you know, do those things for you that like, you can't force, I think, in like a network, in like a more, you know, over networking way. Like, I think if you think about networking as go make nerdy friends, having a few really good nerdy friends will take you so much further in that space than in any other way.
I agree so much that I gave a talk called why you should stop networking and start making friends. So I'm going to shameless plug for that one. It's very short. But I wholeheartedly agree. And I really hope that if you are not on the data science discord server that Posit runs, that I help run for Posit, you should definitely get there. And I'm going to share a link so that everybody can hop on there if they would like to. I'll get that in one second here. But I wanted to make sure that I said, Tan, you really have to go add your dad jokes generator to the only the silliest projects channel on that discord server, because Dami Pak requested a channel for only silly projects. And it has been a joy to go through and look at the stuff and especially the stuff that Dami puts up is just so, so, so fun. She's not here today, but go find her on the server and look at her silly projects. She just wrote something that was like a model that would let her predict when that Lou Bega song came out based on the names of the women.
And like when their names were popular baby names, which cracks me up. Mama number five. Thank you, Rachel. I'm like the Lou Bega song. It is so freaking funny. So go check out only silliest projects. Check out what Dami's doing. Put your own silly projects there. Say hello to Tan who was there on that server with me. And I really, you know, can't wait to see you. Also, if you get on there in the discord server, I only know your name from Zoom as your real name. So if you get on and you have like a username, consider changing it to your real name. So I know who you are and say hello to me. And if you come from data science hangout and you are all in the server, tag me in the data science hangout channel and I can make your name yellow as part of the hangout crew. I'll be making Tan's name yellow in just a minute because he's now a data science alumni.
All right. Thank you so much, Tan, for joining me. This was really fun. I really appreciated you hanging out with us. I hope you had a good time. Thank you. It was awesome. Super glad to be on.
Wonderful. Everybody, I cannot wait to see you next week. We have Dudi Roy, head of clinical data science capability management at Boehringer Ingelheim Pharmaceuticals. That's going to be a great time. I really, really hope that you will consider working out loud. Consider making some friends. Consider sharing projects, sharing code, sharing your mess and being brave. I'm very proud of all of you for being an amazing community. If you would like to save the chat because there's a bunch of cool stuff in there, there's three dots in the top right corner of your chat screen and you can save the chat there. This will end up on YouTube in a couple of weeks and all these resources will also be linked in the YouTube video, but you can also ask for resources in the Discord server and we will share them there too. Okay. Thank you for hanging out with us. We'll see you next week, everybody. Have a good one.