Resources
Apr 28, 2023 |
59 min
|
849 views

Emily Riederer @ Capital One | Explicit design at the start of a project | Data Science Hangout

We were recently joined by Emily Riederer, Senior Manager - Customer Management Data Science & Analytics at Capital One.

We discussed how a strong foundation in high quality data infrastructure and reproducible tools sets the stage for innovation in modeling, causal inference, and analytics, and so much more.

Diving into a question asked at (42:38): What is your thought process for solving a problem that you don’t know how to solve immediately?

One thing that I think is a really undervalued part of that process is thinking about how you will know a good solution when you find one? Also, how would you know if there was a good solution staring you in the face and you already had it?

I think the more unstructured and complicated a problem can be, it can almost be a little deceptive of what's good-- which can have one or two bad outcomes.

You find a good solution, but you don't realize it's good so you keep going
You spend a lot of time chasing after an outcome, and only then do you realize, I solved the problem I was trying to solve but it wasn’t the problem I wanted to solve.

Something I've really been experimenting with in my own work is having a lot more of an explicit design stage at the beginning of a project and thinking, how can you do a pilot?

If I'm trying to predict some target, can I take those two values of that target and plug them into a downstream problem I actually thought that I was going to solve, and make sure that's actually what I want to solve?

Almost like frontloading model evaluation with even a fake solution is the first step versus last step.

Then, I'll check on one other point.

I think the other aspect of that – going back to that level of abstraction – is figuring out how to take the context out of my problem to make it something more Googleable.
So I mean thinking, not being like, “oh, this experiment, the random seeds were wrong, so I don't have a control population - what do I do?”

Backing that into more of a general question - “how do you sample a synthetic control through observational data?” which is something you can Google and then find a ton of resources about.

I think pushing myself on what I want, and then finding the right framing at which to ask for help.

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc

To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Welcome to the Data Science Hangout. Hope you're all having a great week. I'm Rachel Dempsey. If we haven't had a chance to meet yet, I lead our pro community at Posit. And so the Data Science Hangout, if you've never been before, is our open space to chat about data science leadership, questions you're facing, and getting to hear about what's going on in the world of data across different industries. And so it happens every Thursday at the same exact time, same place. So if you're watching this recording sometime in the future on YouTube later, the link to add it to your calendar will be in the details below. And we'd love to have you join us live too. Together, we're all dedicated to making this Hangout a welcoming environment for everybody. And so we love hearing from everyone, no matter your level of experience or area of work. It's totally okay to just listen in here. And there's always three ways you can jump in and ask questions or provide your own perspective too. So you can jump in by raising your hand on Zoom. You can put questions in the Zoom chat. And feel free to put like a little star next to it if you want me to read it instead. If you're in a coffee shop or kids are running around or whatever is going on. And then third, we also have a Slido link where you can ask questions anonymously. And I see Hannah just shared that in the chat right now. So thanks, Hannah.

I will say, Emily, I have wanted you to join us here for so long. And we've also had a number of requests to have you on. So I'm so excited to have you here with us today. Emily is Senior Manager of Customer Management, Data Science and Analytics at Capital One. Now, Emily, let's maybe get started by just having you introduce a little bit about your role and what you do and maybe something you like to do outside of work too.

Yeah, definitely. And thank you so much, Rachel. I'm thrilled to be here. I've benefited so much listening in, watching the videos from all these conversations and really excited to take part today. So I've been at Capital One for about the past seven years. Over that time, I've held a number of different roles. But fortunately, both the data science toolbox and specifically R has really been like a through line for me throughout whatever I've been doing, spent some time on in the more strategic analytics and modeling spaces, solving kind of very specific business problems related to our card business. I'd say overall, I think my career has pulled me kind of like progressively further and further up the data stack is no matter what problem downstream, I'm really motivated by solving. I always realize there's some roadblock just one level higher. And I get really excited about like, well, how do I just solve that one pain point and that one? So after starting out more in that first line data science space, moved a little bit further upstream, when I got really passionate about inner source tooling, and bringing the best of what I saw from the outside, our community into my company, and building internal packages to help us solve common business problems like customer lifetime value modeling, setting and forecasting KPIs, and even connecting to just different enterprise systems. And really enjoyed kind of like building out both those packages, but also really figuring out that you needed to also build out an inner source community around them. Again, copying so much from what I've learned from hashtag our stats, Twitter, and then kind of continuing that trajectory upstream, moved more and more progressively into also adding in elements of analytics engineering and data engineering to tackle kind of standardizing, harmonizing, breaking silos, and that raw data input that would then flow through the tools and the ultimate data products. So that's kind of my trajectory and where I've landed today.

Awesome. And what's something you like to do outside of work too? Big runner and big reader. So nice. We'll have to ask you about some favorite books that later on too. But I love that you just started off with talking about inner source there, and that's something that actually came up on a Hangout, maybe that was last year, was Zach Garland at MasterCard. And I'm curious for all of us here too, like what is inner source really mean? And can you explain a bit about what that community looks like at Capital One?

Inner source at Capital One

Yeah, definitely. So in some ways, it's almost a funny thing to explain because I think to the open source community, it sounds so natural. It's almost like, why does this concept require its own name? But I think sometimes like in large corporations or just in different environments, you don't just have that nice synergy where someone sitting halfway in the world as you has happened to already solve your problem and put the answer up on GitHub. And so there can be a lot of, I think, redundant work, both in one person's project, like moving from project to project, or different teams solving like very similar problems, maybe like sometimes reinventing the wheel on even the framework or solving that problems. Other times, like using the exact same framework, but like rewriting the same boilerplate codes, like for hyperparameter tuning or something like that. So inner source is just really the idea of kind of both building kind of tools and an internal code base that multiple teams can contribute to over time. And as I mentioned, like that only also works if you like try to build up that same sort of community and enthusiasm and inclusivity and knowledge sharing around it to support it.

And so that's with like a lot of the internal packages that you create too?

Yeah, definitely. So yeah, a lot of, I think you can like think about it, the definition pretty expansively, you know, I mean, being anything from data, you know, I mean, like, there's so many opportunities, I think, standardized across the data stack. But I think the place I've definitely focused on it most is in the art package space. I love that. So, so hopefully, like one package for one thing, and then everybody around across Capital One is using it for that.

Yeah, that that's always been, like, been the goal. And I think the other really cool thing about that is really digging into like, the levels of abstraction, and that like, not like, you obviously never want to constrict different teams from like solving problems in different ways, because different parts of the business inherently should be customizing and tailoring. But it becomes a really fun problem thinking, you know, what should be standardized, like, not everyone wants to write the glue code to like connect to database and get through a proxy. But where are the right degrees of freedom of like, one optimization method might be truly better for one type of problem than another. And then it also kind of, I think, helps you aggregate those business use cases, and create that kind of body of knowledge of like, these are how different teams like, have solved this different problems, why they needed to like layer in different tools. And like, it becomes kind of a knowledge store, I think, for some of the complexity of any given problem.

I'm always really interested in, like how you can standardize across standardized things, but also share new packages or share ways of doing things across such a huge company, too.

Yeah, no, I mean, I think definitely, as best you can, I think, like, a lot of companies do use like a version control system, like GitLab or GitHub. So like, obviously, first step minimum bar is just like getting it out there. But I think there's, there's a great little O'Reilly book on InnerSource, that one of the chapter headlines is literally like, just because you use GitHub doesn't mean you're doing InnerSource, which I think is really critical that like, I think in open source, we are sometimes spoiled by the extent to which it is, if you build it, they will come. They will not always come internally. I think like corporate incentives are different. Sometimes the culture of people just have no expectation that if they look, they might find something. So I think definitely, there's a little bit more of shoe leather beat the pavement in terms of talking to other teams, trying to go presented internal forums. But also, again, building up that community with things like, I leaned very heavily on like Slack channels to kind of like bootstrap and internal version of like Twitter, to try to get those just like, serendipitous say sync conversations going.

if you build it, they will come. They will not always come internally. I think like corporate incentives are different. Sometimes the culture of people just have no expectation that if they look, they might find something.

Yeah, definitely. Libby, I see you had asked something in the chat a bit earlier, do you want to jump in?

Sure, yeah. I was wondering if you could describe in general terms, without giving a bunch of details that you can't give, maybe some of the projects or the problems that you are working on solving with your team right now?

Yeah, definitely. So I think pretty wide range of things and I'm trying to like, figure out how to answer that, be like perfectly honest, it's like the right level of abstraction. But I think something that's really been top of mind for me lately, is just like, how you can really shape that kind of like raw data layer, and make it like the most usable for kind of like the down last mile analysis. So, one broad area of interest that I'll say has been really on my mind, is thinking about how to structure raw data into like really meaningful, you could almost call it like a metric smarter feature mark, although it's not although it's not quite the same thing, but just like how you can embed kind of like more like context about a business problem in individual data elements, to have like more of those off the shelf for use kind of across, you know, metrics tracking across like, modeling and kind of like feature engineering. Some of the like business areas that I'm most focused on these days, in our card space is our kind of like credit lines, and select practice strategies to help be sure we're like, you know, like equipping our customers to like, have the best kind of both like financial health, but ability to like pursue their interests.

Thank you. I'm starting to see a lot of questions over in the chat too. And Mike, I swear I did not see your question there when I asked that. So maybe I'm starting to think like a data science leader a little bit more here. But Mike, did you have any follow up questions or anything else you want to get out of like communicating inner source packages and best practices? Yeah, thanks. I mean, I, first of all, it's great having Emily on the call. I mean, it's great having Emily on the call. First of all, it's great having Emily on this data science hangout. I've been following your blog posts for ages, and they're really inspirational. So please keep doing those. And the question was really about that. What I see a lot is that when you send an email or send a message on Teams, or have a community meeting, maybe you only get 20% of the whole population of people you're trying to target at that one thing. And, you know, there's value in spamming people across many platforms. But I also know that many of the people that I want to try and target with, here's a best practice, or I've written this document, or here's a package you might be interested in. Some of them have already kind of muted that channel, or, you know, got an email rule that just, you know, I'll tuck that straight into this folder. So how do you kind of gradually increment the number of people who see those messages?

Yeah, I mean, I think definitely the like, in an enterprise setting, I think I have also loomed more also on those, like, personal relationships to get the word out than you would necessarily in the open source context. So both in terms of, I think, kind of like leaning on leadership to like, help identify people that may be working on similar projects or patterns, kind of throughout the company. Or ideally, I think having a forum, where you're intended to kind of like, share updates, not on an individual project, because it's like, how do you get people to come to that forum initially. But if you can get people interested in the idea of like, technical knowledge sharing at a higher level, aggregating kind of like, many types of information in that one space, I mean, again, to like, make an open source analogy back, like kind of like, internal, our open sci analog, or internal, like our weekly type newsletter, I think it can be easier to like, sell and socialize the broader concept than maybe an individual like, artifact thereof.

Yeah, thanks.

Thank you. Marlee, and I see you had a question in the chat. Do you want to jump in here?

Yeah, sure. Hey, Emily, by the way, great presentation at TinyComp. That was where I first got to see you. So that was fantastic. And my question was, what do you do in your job that you didn't actually anticipate seeing yourself doing? Like, I think we all kind of get blindsided

by this weird, like, niche skill or task. And we're like, I didn't sign up for this, right. But I was just curious, because you've been at Capital One for a bit. Do you have anything like that you could share?

Oh, my, like, I, if anything, I feel like that's truly the story of and to your point, like, I don't think I'm unique in this regard. But I think, um, does almost feel like it summarizes my career arc, went from like, math to statistics, because I wanted to like, math for the real world, but then found like, the shift from like, deductive to inductive reasoning, like, not at all like math, then went from stats to data science, because wanted to, like, use the statistical toolbox, to, you know, like, solve these, like, beautiful probabilistic type problems, but then found, like, all of my challenges for more into like, data and tooling layer.

And so, I mean, I think it's a, this may be a kind of generic answer, but I think, like, being obsessed with, like, not only like data quality, but even upstream of that, like, how is data getting collected? How do the source systems work that are processing the customer records that are then spitting out the data? I think, like, I definitely never expected myself to, like, keep going, like, further upstream and further back into more of a like, technical stack, as opposed to, like, proving a nice little theorem, which is where I started out.

But I think definitely something where I think a lot of my mindshare goes, but that I find is, like, super helpful for the last mile. Because even if you know some, like, really weird perk in your system of, like, oh, we were never able to, like, email these customers for x reasons, it's like, suddenly, then you also know, like, population that, like, something happened to you for a systemic reason. It could feel some, like, really interesting analysis.

So, I think the, both going upstream, but then being able to, like, round trip back downstream are probably two things that I never would have anticipated.

Yeah, cool. It's funny you share that. I recently was told that data science were snobs, like, we're only happy with the tail end of the data stream. I quickly had to kind of, like, snap my head around and say, well, I think we're, if anything, more holistically, we're interested in the holistic health of data, like, from start to finish, literally nurturing it from its inception all the way to the end. But I found that really interesting. So, it's funny that you're reiterating that, that you find yourself looking downstream just as much as you are upstream and where you're at. So, anyway, thanks for sharing. I'll shut up now.

Not at all. And yeah, I love that. I mean, to me, I think that's also part of what really puts the, like, science and data science is seeing the whole system. I forget, someone in the more ecology side of our stats Twitter once commented on something I wrote about, like, oh, like, the data generating process, like, I have all these same concerns as you do. It's just, for me, it's when I fell out of a boat and I dropped my notepad. And it's, like, very different, like, cause of missing data, but potentially similar outcome.

when I fell out of a boat and I dropped my notepad. And it's, like, very different, like, cause of missing data, but potentially similar outcome.

I love that, putting the science in data science.

Alan, I see you had a question in the chat, too. Want to jump in next?

Yeah, sure. Hi, Emily. I'm really curious to learn a little bit more about where your team fits in, like, if there's a bigger analytics organization in which you fit, and do you have peers who are similar leaders of, like, analytics within other functions and teams? And if you collaborate and what that expectation is, or if your focus is mostly kind of inward to your team. And I know it's probably a mix of all of the above. I'm just really curious in sort of, like, how your team fits in the rest of an organization that I imagine is really big.

Yeah, no. And I'll definitely, again, like, dance around that potentially a tiny bit, just to not talk too, too much about Capital One specifics. But, like, yes, super large organization, very matrixed. So, and, like, I've always sat kind of, like, within a specific line of business, and been predominantly focused on solving that business's problems. I was pretty lucky to have, like, super supportive leadership, I think, early on and continuing throughout my career that really kind of, like, latched on to, like, a lot of these same ideas of abstraction and scale, and was willing to kind of, like, support building out a team that honestly is, like, a little bit more kind of, like, business facing and interdisciplinary skill sets than would usually even be tackling some of these problems versus kind of, like, a pure tech team or data engineering team, or breaking all those pieces up. But, yeah, we definitely have a lot of great kind of, I think, analogs throughout the business that have their teams structured, really, in many different ways, you know, like, the mix of staffing, whether it be more business focused people or engineering focused people, whether it be, like, centralized or federated. And it can definitely be complex navigating a really large org in that way, but I think it does, like, help bring a lot of interesting ideas to the table when you have, like, different types of people in different, like, org configurations, all trying to, like, tackle problems, bringing the best to bear of their own tool set.

For sure, yeah. And your journey, it sounds like, has been one of building that really focused team in part. Yeah, definitely. And I think the, based on kind of, I think, the biggest bet that the people that were maybe, I mean, like me starting out a little bit more downstream, maybe not the best state, like, not the best engineer by mile that you could, like, put on that work, that kind of, like, finding people that can, like, bridge those multiple worlds and kind of, like, pull that downstream context upstream, I think is maybe not, not what I'd say is the overarching norm, but I think it's, like, definitely a pattern that is gaining some, like, good, good traction. Cool. Thank you. Thank you.

Book recommendations

I see Javier is calling out your amazing book collection behind you. And so, Javier said it looks like data science book heaven. Do you have a favorite book behind you?

Oh, my goodness. That would be a hard one to pick, and we could use the rest of the hour going through those. One of my weirdest, or, like, strangest, like, post-pandemic hobbies has been, I had the opportunity to review a ton of books for, kind of, CRC Press, and then, as a consequence, have kind of built out my library with a lot of those. I think off the top of my mind, like, obviously, like, the Gallery of History Thinking Statistics, I think, I know, is a little bit of a generic recommendation, but if, I think, can just be, like, such a lovely, like, whether or not you're Bayesian or Frequentist, like, won't get controversial here, but it's just such a, like, beautiful probabilistic approach to thinking about data and modeling. I'm very, I have a lot of epidemiology books just because I, like, never was really in the health sciences field myself, but I've found those to actually be some of the most informative, strangely, for business use, because I think stats books often talk about things from the standpoint of, like, assume you have this perfectly clean data, it's in this matrix called X, now we'll prove some nice theorems, whereas epi books almost start more with, like, 5-1-1, we want to figure something out, we don't have any of the data we want, there are biases everywhere, what do we do given that? And I think, you know, like, again, when you're just, like, trying to work with a lot of larger observational data in an org setting that's just, like, super pragmatic, what can I do with what I have available is pretty cool, but, again, you could, like, definitely pass along a lot more specific recs based on the category.

What was the title of the first one you recommended?

I may have said it wrong, I think it's actually Statistical Rethinking.

Okay, if anybody wants to share the link to that in the chat, that would be super helpful too.

Benefits of contributing to open source

But Emily, somebody, something I wanted to ask you is, I know so many people here in our Hangout room have benefited from your blogs and conference talks and so much more, and I was just wondering, like, for you, what have been some of the benefits of contributing to the open source community? Oh, my goodness, so, like, so very many. I think, first and foremost, obviously, it's just, like, all of the amazing, like, relationships you form, people you get to meet, and being a fly on the wall of, like, conversations that, like, I'd never get to hear, I'd never get to hear, like, epi professors talk, or biostats professors talk, or econ professors talk in my day-to-day life, but just being able to continue to, like, pull, and I think this is especially unique to R, from all these, like, really rich interdisciplinary communities and steal the best ideas, and then take them back to my day job, and then I look like I, like, came up with a new idea, and I didn't. I'm just stealing like an artist, so I think that's, the people is definitely a huge benefit.

In terms of, like, contributions, I think something that I found a very surprising benefit is the way that it's made me have to think a lot more critically and a lot more formally about what I was doing anyway and otherwise. I think even as I've kind of dodged a couple of questions here today of, like, oh, can I, like, how much detail can I go into here? The upside of that is, I think, a lot of the things that I found that I can, like, really lean in to talk about, it forces me to, like, think about the problems I'm facing at that higher level of abstraction and think about, like, what have I learned from this that's not just true to my last project but is true to my next project and is true to other people's projects? So I think it's definitely a fun intellectual challenge, I think, almost like taking a final exam in school to kind of, like, force you to formalize your own knowledge.

That's really interesting. Yeah, so, like, by not being able to give some of those specifics, you're able to then, like, take a step back and look at this big picture that can then help other people, too. Think a little bit more in terms of those, like, core patterns. Definitely.

Advice for building a data science portfolio

I see, apologies if I mispronounce your name. I'm not sure if your first and last are connected there, but Hiba Bukhari, you had a question into the chat. Do you want to jump in?

Hi, Rachel. That was close enough. Yeah, hi, Emily, very nice to meet you and, yeah, great talk. And, yeah, great talk. I love your blog. And I was wondering, for someone who's making the transition into the data science world and trying to build her portfolio and her skills, I think, from what I'm seeing, is more important than the experience is showing off the skills and showing off the personal brand. So, what advice would you have for someone who is, like, I'm still building my blog. I'm still trying to create stuff to be able to put them out there. What advice would you have for me or someone like me? Yeah, no, great, great question. And I think, I'm trying to remember, I think it was back in, like, 2018 or 2017. It's been a while, but David Robinson gave an amazing talk at RStudioConf about this that's probably floating around in a video somewhere that I know I personally found really empowering. I think, kind of, like, the first and foremost advice, I think, is, like, get it out there. Like, anything that's out in the world, I think, was really the thesis of his talk, is better than anything that's on your hard drive.

get it out there. Like, anything that's out in the world, I think, was really the thesis of his talk, is better than anything that's on your hard drive.

And I think that, you know, like, can at times be intimidating, but I think, a, I think, like, a lot of work can be, like, a lot more impressive and interesting to the outside world than sometimes we even believe when we've been, like, mildly working on it for so long. And also, especially in a great community like the RStats community, I feel like people are really supportive that, like, yeah, they might have some advice or other ideas, so maybe they leave a comment, but everyone does it in such a genuinely nice way. Like, if anything, sometimes I, like, crave for somebody to, like, comment on something that I wrote and be, like, no, you're wrong, and here's 10 reasons why, because I think that'd be a really interesting discussion.

But I think, like, a, definitely don't be intimidated to get stuff out there. And then, I think, second, like, kind of, like, key thing is, again, like, finding projects that you're, like, genuinely interested in and passionate by. You know, I think a lot of people, like, there's decreasing marginal returns from, like, an employer's perspective, seeing someone else play with, like, the Titanic data set, whereas, like, I think you'll

both showcase your own skills better, as well as kind of, like, really engage your audience, if you just, like, kind of chase after some passion and find something, like, a little bit new and different to bring to the table.

Thank you. I think I missed one of your questions from earlier, Travis, and just want to have you jump in here. Sorry about that. I think that maybe was a while ago. Yeah, no, it's fine. Hey, Emily, sorry to break your level of abstraction that you filled up here, but I wanted to ask you about dbt. I know we talked about this a couple years ago. I wonder how that's evolved for you. Are you using it still these days? Have you integrated more with R? You have the dbt flyer package, which I see, like, kind of is just hanging out there, trying to generate interest across the, across the R community. Like, you know, it's a really cool complimentary tool, but not a lot of integrations yet, as far as I can tell. How's it going for you?

Discussion on dbt and R integration

Yeah, no, I'm still, and, like, maybe I'll just take a step back and, like, explain for a minute, like, kind of what dbt is for those that aren't familiar. dbt is a framework for building data pipelines in SQL, and what it kind of nets you is a lot of the same benefits that I think we all value with, like, reproducibility in R, which tends to be harder with databases and with dbt, so you can do things like DevPad deployments, you can do better testing, it helps you modularize your code better, so you can, you know, have easier version control, easier code reviews, and it just, and with some ginger templating, it makes SQL more like a higher level kind of programming language with legitimate control flow. Definitely, if you work in, on databases, I'd recommend checking it out, because I think there's, like, a lot of spiritual similarity with just things this community, like, tends to value and get excited by.

I continue to be, like, kind of big fan of using both tools. I'd have to say I've never really found that sweet spot of exactly learn how to integrate them. I think, still tend to have, like, dbt pipelines running more in the database layer, but kind of switching more into R mode for real, obviously, like, kind of analysis, reporting, modeling, etc. I've always felt like there had to be some, like, linkage, especially with something like dbFlyer, but haven't, like, have not gotten there yet, and would be, like, fascinated if anyone else has, like, played with both and has any takes there. Yeah, likewise, in all your spare time, right? And I'm sure everyone else is in the same space, but yeah, my experience has been the same. Yeah, I feel like when I'm in dbt, I'm in straight SQL world, and there's no direct linkage, but mirroring your thoughts on the spiritual similarity, the community there is really, really good, and yeah, it's a good tool. Yeah, and it, like, minimum, it's, like, it helps kind of, like, keep your mind in the same place, even if, like, there's not quite that, like, deep, deep integration.

Sounds like this would be a good part to share with the Tidyverse team, too. Yeah, let's.

What has kept Emily at Capital One

So, I just want to remind everybody, if you joined a little bit later, that you can ask questions a few different ways. So, there's a Slido link if you want to ask anything anonymously. You can always raise your hand here on Zoom, or you can continue putting questions into the Zoom chat, too, and if you put a little star next to it, I'll know to read it out loud, but I see Toyin had asked, piggybacking on Libby and Marlene's earlier questions, what has kept you at Capital One for seven years? That's a great question. I know, especially in this day and age, I think it's, like, the less common route to stay at a, like, single company for a long time. I think, definitely, like, I think pros and cons to it. I think definitely the thing that's kept me motivated is being able to, like, continue to, like, expand my scope, expand my skill set, both in terms of, in such a large company, having different areas of business to move around to, which is, so I'm still kind of, like, thinking about different, like, strategic issues throughout my career, somewhat similar to, like, changing companies, as well as kind of growing out my technical skills in different dimensions. So, I think I'm definitely somebody that, like, gets very edgy if I don't feel like I'm continuing to learn, but I think I've found ways that, like, I really feel like I continue to be able to do that in my career.

And secondly, this is a very, like, silly reason, definitely not a career-defining one, but going back to InnerSource, I will warn you, like, packages are kind of like children, and there have been times in my career, even where I've been offered a different role internally or something, and I've, like, been like, oh, but then I've had to, like, I don't know how to say goodbye. So, be warned.

AI's impact on data science

Hi, everyone. Good day. Yeah, so I've been following this space very carefully, and I've been a part of several discussions about the impact that AI is and will have on our profession. I must say, I'm just uncertain about the future, to be honest. It's developing and evolving so fast, you know, so I was just trying to understand, from your perspective, what do you see, how AI is actually impacting your current workflow, and what your thoughts are in terms of how it will impact the data science space in the mid to, well, near to mid-future.

Yeah, no, that is such an interesting issue right now, and I feel like definitely, I know a lot of very strongly opinionated, like, strong opinions on both sides of the spectrum of whether it's going to change everything or nothing. I think through the type of work I do, and the type of, like, kind of values, per se, that I have about data science, I think there are, like, large parts of it that I don't expect it to change a ton, or I almost hope it doesn't change a ton, and I'll give, like, a couple of examples of that. I know, like, I think even there are a lot of companies and tools right now that are working on, like, automated NLP query generation, which, on one hand, like, seems really cool. Yeah, I'd love to be able to just answer anyone a question, ask a question and get an answer, but then there's, like, I think the data quality, like, part of my head, like, like, the fire alarm goes off, that, like, if, like, there may be an AI that's, like, smart enough to, like, write a sonnet far sooner than there is one that's smart enough to think, based on the kind of conditional probability this person wanted, should I cast the nulls in this table to a zero, or should I leave them as zeros, and what should be in the denominator?

And, you know, I mean, I think, if anything, like, data quality and data bias will save us, because I think those are so sufficiently hard for humans to ever be precise enough in language about what they mean, and I'm kind of, like, clear enough and what the right result should be, that I feel pretty confident that's going to remain a, like, solidly, like, human problem for a much longer time. And then similarly, on the last mile, like I mentioned before, I think I'm very passionate about kind of, like, thinking about feature engineering, curating, like, both automated solutions, but also, like, kind of understanding and bringing, like, context into data science, whether you even think about cost-sensitive loss functions, like, model evaluation beyond traditional metrics, and I could imagine kind of, like, more DS work shifting to those, like, first mile and last miles of, like, data quality and curation, and kind of rigorous business-focused evaluation, but I definitely don't expect their jobs to, like, go away because of AI.

data quality and data bias will save us, because I think those are so sufficiently hard for humans to ever be precise enough in language about what they mean, and what the right result should be, that I feel pretty confident that's going to remain a, like, solidly, like, human problem for a much longer time.

I understand what you're saying, but even a lot of the processes that you talk about, we see AI making its way into that space. I understand and I totally agree with your point about, for example, the feeling of data, right, and an AI system may not know the context of what a missing value should be or why the piece of data is missing, but my thought process is a little broader in the sense that, for example, someone asked earlier what you should be thinking about when you're entering the data space, as in, if you're in school now, you're in maybe university, you're in the middle of doing a data science degree, you know, what will your future look like? So we have seen, for example, in the last maybe five to ten years, last five years, we have seen where, like, data science is, like, the hottest topic, you know, the hottest job, right? Do we think that this hot streak is going to continue into the future, or will this hot streak be replaced by AI, if you understand what I'm asking?

Yeah, definitely. I mean, at the end of the day, I think they're probably all just tools in a toolbox to solve, like, super wide range problems. Like, I don't even know if, like, data science will always be a career, so much as, like, data science is, like, kind of an expectation of, like, something professionals should be able to apply to the problems they have at hand, like manipulating an Excel spreadsheet. And similarly with AI, I think there are probably some things, like, in a legal setting, like, parsing a ton of documents, generating some hypotheses, like, helping maybe structure a preliminary screen for, like, legal discovery or something. Like, I think there's some kind of assistive things AI can help with, but I, like, I personally, I think, like, don't love the idea of a world where it's every, like, making the final decisions, so I think plenty of space at the table for both.

Yeah. Okay. Okay. All right. Thanks. Thank you. I love that perspective on it and, like, thinking about how, like, everybody knows how to use Excel right now. Maybe at some point in the future, everybody will know how to do some level of data science.

Solving problems you don't immediately know how to solve

I see, Jonathan, you had asked a question on, there's a star, so I'll read it, but it's data science involves tackling complex problems. What is your thought process to solving a problem that you don't know how to solve immediately? Like, what resources do you tap into that assist you with problem solving?

That's a really good question and a really, like, prudent question lately, I think, for a couple of different reasons. I think the, one thing that I think is a really undervalued part of that process is thinking about what is, like, how will you know a good solution when you find one, and how would you know if it was a good solution if it was, like, staring you in the face and you already had it? I think the more unstructured and complicated a problem can be, it can almost be a little deceptive of what's good, what's good, which can have, like, one of two kind of, like, bad outcomes of either, you find a good solution, but you don't realize it's good, so you keep going, or you spend a lot of time, like, chasing after an outcome, and then you get there, and only then do you realize, like, I solved the problem I was trying to solve, it was not the problem I wanted to solve.

So, something I've really been experimenting with in my own work is having a lot more of an explicit, like, design stage at the beginning of a project, and thinking, like, how can you do a pilot? How can you, like, you know, if I'm trying to predict some target, can I take those, like, true values of that target and plug them into a downstream problem I, like, actually thought that was going to solve, and make sure that's actually what I want to solve? And almost kind of, like, front-loading model evaluation with a, even, like, fake solution is, like, the first step versus the last step.

something I've really been experimenting with in my own work is having a lot more of an explicit, like, design stage at the beginning of a project, and thinking, like, how can you do a pilot? How can you, like, you know, front-loading model evaluation with a, even, like, fake solution is, like, the first step versus the last step.

And then I'll attack on one other point, and sorry, I'm kind of rambling here, but I think the other aspect of that, just, again, going back to that levels of abstraction, is figuring out how to take the context out of my problem to make it something more Googleable. So, you know, I mean, thinking, like, not being, like, oh, this experiment, the random seeds were wrong, so I don't have a control population. What do I do? But then, like, backing that into more of a general, like, how do you sample a synthetic control from observational data, which is something you

can Google and then find a ton of resources about. So, I think pushing myself on what I want, and then finding the right brain at which task for help.

That's great. Thank you. Anas, I see you had a question. Do you want to jump in here next?

Sure. So, Emily, I'm sure you've been in this position before, but any tips and advice on communicating to non-technical stakeholders and or leadership?

Communicating with non-technical stakeholders

Yeah, no, that's always a hot, like, enduring hot topic, and for very, very good reasons, I think. I think, like, a couple of different, I think, approaches that I tend to use there is, A, like, leading with the solution versus kind of, like, leading with the solution, leading with impact versus leading with that, like, kind of process. And I think as many people know, I'm a, like, huge R Markdown fan, and maybe now I should say Quarto, but I've actually, like, come to the realization that I kind of need to even change my workflow for working with a tool like that because, like, inherently, like, an R Markdown workflow is kind of very linear of, like, I cleaned my data. I did the analysis. I got the outcome. But then from a storytelling perspective, I think you can really, like, get the hook and get the buy-in to, like, keep that conversation going if you lead more with the outcome of, like, this is why what I'm about to tell you is important.

And then some people want to really dig deeper and understand the mechanics. Some people won't, and I think it's also as, like, data person learning to accept that that's okay that they don't, they don't want, like, like, I think all of us, it's very, like, it can really be about the journey versus the outcome, but accepting for, like, leaders, reasonably, it is often about the outcome. But then when I want to go into those details, I think I both either try to, like, lean heavily on both and or either of, like, metaphors and also, like, diagrams or one really compelling plot, just anything to make it more tangible versus, like, purely conceptual.

Thank you. Thank you very much. I see, Lewis, you had a similar question as well. Do you have any follow-up questions or did that answer you as well?

I guess maybe a follow-up and maybe it really is the same, just to clarify. I guess that in one sense makes it sound a communication kind of piece, but do you have the same answer if it's a brand new project or maybe a completely different approach? I don't mean a new quantitative approach, but maybe here's a stakeholder who's not really worked with quantitative people or maybe is wedded to a tool set. Maybe they use a business intelligence tool and you're advocating Shiny and it's interactive or something like that. So, to them, there's some kind of context, which even if, let's say, quantitatively there isn't, do you have any advice on how to advocate that and get the buy-in? And, of course, I imagine it's probably still a question of, here's why, what I'm about to tell you should be interesting or helpful, but do you have any anything else that's come up where you say this has really worked when I'm trying to advocate something new?

Yeah, no, I think that's a great point of tools versus analyses because tools are so inherently abstract, it can't make it harder, but I do think probably how I've translated it, leaders inherently, I think, care a lot about the people on their team, hopefully from the genuinely caring perspective, but at least from the managing capacity perspective, bare minimum, and I think kind of leading with the case, I think most tools, you can also bring a human dimension into that story, whether it's such a pain point that people have to, they're doing this manual work every single month, then that's inhibiting both them doing more interesting projects in their career development, but there's a way to automate this thing, or like, hey, three months ago, we had this error and this breakdown in this process, and that's because we don't have good version control on GitHub, and I think always, you can tie it either to business outcomes or people outcomes, but I always, I think, try to anchor more to what does any tool do for you.

And there's a broader framework that I really like for thinking about this called the jobs to be done framework that kind of comes from the product management world, and essentially what that says is you can think about a tool as someone you're hiring to do a job, and I'm kind of saying it at a trivially high level, I think there's more to the actual theory, but I've always liked that, kind of framework for, like, thinking about the interview, thinking about, like, why should I, like, bother to, like, hire and onboard you, and is this tool more, like, almost going back to the AI discussion, is this, like, an intern that I only want to use with a very high level of supervision, or if you're thinking about CICD and process automation, it's, like, am I hiring an executive or a contractor to just go run this process for me in an abstracted way, and I think that's, like, that framework, I think it's really kind of, whether it's implicit or explicit, helped kind of my, like, storytelling around those technical tools also. Very cool. Thank you. Thank you very helpful.

an executive or a contractor to just go run this process for me in an abstracted way, and I think that's, like, that framework, I think it's really kind of, whether it's implicit or explicit, helped kind of my, like, storytelling around those technical tools also.

Thank you. So I see we have a few minutes left here, and I think still quite a few questions here, so I'm going to go and try and go through some of these quickly. But I see Bill had asked, he said, I'm in pharma, but wonder if there are trends, blessed by management, to move away from some alternative software, which is how Bill says SaaS, but maybe more broadly, like, are there trends from your leadership, like, moving towards open source, or do you have free reign over the tools that you get to use?

So I'm, like, definitely feel really lucky to be in an environment that's, like, pretty much always been very, very friendly to open source, and I think that is, like, I know a trend I've seen across the industry. I've had the opportunity also to do some consulting projects with other people, more into biostats and pharma fields, taking that leap to, I'd copy Bill's very diplomatic phrasing if I could, but other proprietary options to the R world. And especially, I mean, I can only imagine, I think, the intersection of data tooling with the current, like, economic conditions, I have to imagine can only, like, kind of accelerate that trend when it's becoming an increasingly, like, easy to hire skill set, much more attractive option through the bottom line. And I think that gains that have been made in adoption in both, like, banking and finance, as well as, like, healthcare, really put a lot more of a rubber stamp of approval on some version of open source in really highly regulated industries.

Advice for transitioning into data science

Thank you. And Amrit, I see you had a question about transitioning into data science roles. Do you want to jump in here? Yeah, thank you, and hi, Emily.

Yeah, I, so my, like, basic question was that I'm just transitioning into data science roles, or I'm looking for data science roles. So I was just wondering, are there any considerations I should be having while interviewing with companies and in terms of, you know, maximizing my capacity to learn in that company? Because, like, I'll just give you some more context. I was just talking to one of my friends who is in the healthcare space, and she was saying she's working as a data scientist, but then she was saying that a lot of her time goes into data processing rather than actually having to, being able to get any inferences out of data or being able to do any kind of predictive work. So yeah, I was just wondering if there are any considerations I should be having, or if this is a non-question, I'm really not sure.

Yeah, no, I mean, I think there's three aspects I'd say to that. I think first, definitely, like, can be really helpful to understand, like, a company's tech stack when you're interviewing and the amount of data they have available that you don't want to be, like, hired to be a data scientist and then show up on day one and have them be, like, oh yeah, here's a Google Drive with some CSVs in it, like, go wild. You know, I mean, you want to be sure they have, like, enough data to support analysis and that either you'll be empowered to have the tools to build out the data you need yourself, or maybe they also have, like, data engineering or other job functions.

Secondly, I think it's helpful to be crisp about what you want to learn and grow on in a role. Like, data science can be such a nebulous job title these days. You know, I mean, I think someplace, like, it can mean anything from BI, experimentation, modeling, machine learning, and I think really just clarifying your interests and then just being able to, like, articulate them. Like, companies, like, do not have any incentive to, like, hire you for a role you don't want, so I think, like, kind of being able to, like, share what you're looking for really just helps the matchmaking.

But finally, I think it's also really good to recognize, like, you can learn a ton in pretty much any role, and I think, like, the spending a lot of time on data processing, taking that just as an example, that is sometimes, like, the hard part, the complex part, the part that, like, requires actually still a lot of the data science skill set of understanding how do I shape this data in a way that, like, because I do understand the stats, because I do understand the algorithm, how do I, like, structure this problem in a way that the algorithm can understand the problem? And, like, at the end of the day, I think it's a funny conceit that in school we spend most of our time learning, like, hyperparameter tuning and typing that, like, model.fit, model.predict. That often isn't the hard part or sometimes even, like, the most interesting part, so I think being open with that, like, kind of growth mindset of, like, whatever job you end up in, there's going to be, like, a ton to learn and a ton of really interesting work to be done.

it's a funny conceit that in school we spend most of our time learning, like, hyperparameter tuning and typing that, like, model.fit, model.predict. That often isn't the hard part or sometimes even, like, the most interesting part, so I think being open with that, like, kind of growth mindset of, like, whatever job you end up in, there's going to be, like, a ton to learn and a ton of really interesting work to be done.

Thank you. So I do understand that it's more about the company having a good data culture, and that way I think you will be able to have more chances of maximizing your learning. Yeah, for sure. The culture always matters so much.

It looks like people had some other tips in the chat, too. Like Mike said, a good question to ask is maybe what does a data scientist do on a typical day? And Adam, I thought this was funny you said, a tweet you saw that said, when you're interviewing for data science roles, ask whether you'll have IT or data engineers. If not, congratulations, you're now both. It's too real.

I appreciate everybody sharing the links and resources in the chat, too. I know we just got to the end of the hour here, and I'm sorry if we didn't get to answer everybody's questions. We have one minute if there's any other, like, pressing question if somebody wants to raise their hand in here.

Emily, what is the best way for people to stay in touch with you? Is it through your website or LinkedIn or GitHub? Honestly, I think wherever people are these days, I'm still, like, for now, I'm still on Twitter. I'm still spending far more time there than I should. LinkedIn, GitHub, my website has my contact information, same with my email, like, my handle on pretty much. And, like, I have the fortune of having, like, an unusual enough last name that my handle on, like, LinkedIn, GitHub, Twitter, my Gmail, everything's Emily Reeder at wherever. So, yeah, please, like, don't hesitate to get in touch. Awesome. Like, I just love getting to meet people. Awesome. I just love getting to meet more of the community.

Oh, sorry. I think my internet was delayed for a second, but I just shared your LinkedIn in the chat as well. But if anybody has Emily's website handy or blog and wants to share that there too. But thank you so much, Emily, for joining us today and for sharing your insights and experience with all of us. This was awesome. Oh, thank you. This was a lot of fun.

Thank you.