image: thumbnail.jpg
Four steps for managing data teams | Toby Hall | Data Science Hangout
We were recently joined by Toby Hall, Executive Vice President and CIO at Delta Dental of Michigan, to chat about all kinds of things, including: data science team management, AI governance and the ethical use of data in healthcare, insurance data types and use cases, and hiring for traits vs. skills in data science.
In this Hangout, among other topics, Toby mentions Delta Dental's "guilty until proven innocent" approach to AI. This involves a two-stage approval process that critically examines use cases for ethical implications, data handling (ensuring data stays within dedicated tenants and doesn't train public models), and tool appropriateness. The discussion also touches on the regulatory skepticism towards AI from departments of insurance and the broader environmental considerations of AI use.
Toby also gives loads of career advice, but our favorite was his four-step process for managing a data team:
- Put the right people in the right role
- Give them the right vision of what you're trying to accomplish, ensuring it's clear and everyone is aligned on "what that is and why we're doing it"
- Give them the right culture to get it done
- Get out of their way!
Resources mentioned in the video and zoom chat:
🔗 Posit Conference → https://posit.co/conference/
🔗 Delta Dental Tooth Fairy Poll → https://www.deltadental.com/us/en/tooth-fairy/the-original-poll.html
🔗 TidyTuesday Data Repo (get your reps in!) → https://github.com/rfordatascience/tidytuesday
🔗 Resources for doing TidyTuesday stuff in Python (PydyTuesday?) → https://github.com/posit-dev/python-tidytuesday
If you didn’t join live, one great discussion you missed from the zoom chat was about the Delta Dental Tooth Fairy data, including its outrageously expensive rates and the humorous debate around the inflation of self-reported flossing data. Let us know below if you’d like to hear more about this topic!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
00:00 Introduction
02:28 "Tell us a little bit about your day to day, but also the data teams at Delta Dental of Michigan, what is the type of data that they work with, and what is it that they solve with it, or what decisions are made based on it?"
06:04 "What about third party data?"
07:57 "How do you all decide which language to use and for what applications?"
10:37 "How do you make sure your AI pipeline is accurate in the SQL and the regulation because you are working with health health care data?"
23:22 "What are some tips for leaning into those challenge assignments as the IC, the individual contributor?"
23:31 "How do you identify the challenge assignments? Like, how do you know this is the right thing with the stakes that are right to give to somebody?"
25:54 "Do you share your data as an API or for research?"
36:45 "Coming from a core statistical background, how much would you expect from a fresh data science graduate about statistical knowledge if you were an interviewer for a modeling role? Any specific concepts that you might ask them about in the interview?"
38:17 "How do you make sure your team stays current with new data science tools or skills or methods?"
40:18 "Did you have a ranking of necessary roles that you wanted to hire first, like data engineer versus data scientist or analyst or maybe dedicated reporting staff? And what tools or platforms did you consider essential from day one?"
43:53 "How much of your background in teaching high school students has spilled over into managing your team?"
46:21 "Do you have any advice for how someone could maybe emphasize those traits on their resume?"
49:19 "Do you have a piece of career advice that has either really, really helped you or that you try to give to everybody?"
Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the Paws at Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience. Can't wait to see you there.
I am super excited to introduce our guest today. Our guest is Toby Hall, EVP and CIO at Delta Dental of Michigan. Toby, that is a mouthful. I would love it if you could introduce yourself, tell us a little bit about what you do and what you like to do for fun.
Sure. I'm excited to be here. My name is Toby Hall. As Libby said, I'm the Executive Vice President and CIO for Delta Dental of Michigan. We are actually the headquarters here for three different Deltas, Delta Dental of Michigan, Ohio, and Indiana. We're a little bit like the Blue Cross system where notionally there's a Delta Dental in every state. And we forever have run our entire business on a homegrown IT system. And I lead the team that owns and maintains that. I've got about 500 or so, 510. We kind of swell in the summer with interns. So the report I got last week, I think we have a team of about 510 total. By training, I'm an actuary. That's where I first was exposed to R, spent the first 20 some years of my career doing actuarial work, many of those as chief actuary for Delta. So excited to be here.
And what is something that you like to do for fun? Everybody always forgets that part. Oh, yeah. Oh, good call. So for fun, I have two kiddos. So a lot of time is spent chasing them, going to their different sporting events and school events, reading, and ironically staying up on R, still do a lot of R coding, just hobbyist on the side.
Oh, as do we all. Well, it's wonderful to have you. I will remind everybody really quickly for anybody who just popped in because the waiting room is still going. Hey, waiting room people who just came in, welcome. You can ask questions in Slido for Toby today.
So my first question is usually always a context building question that'll help us ask questions. So Toby, tell me a little bit about your day to day, but also the data teams at Delta Dental of Michigan. What is the type of data that they work with and what is it that they solve with it or what decisions are made based on it?
Yeah, that's a great question. So my day to day, I would say there really isn't a day to day. It's always different. With that breadth of the application that we support and the number of team members, there's always something going on, something different. It's a lot of project management type tasks though. Staying on top of large initiatives, marching towards go live. With insurance, we're very heavily regulated. So there's a lot of data security, things that take up a lot of my time. And then just in the last 18 months, it's obviously been AI is the big kind of buzzword right now. Try and understand how we're using it. Should we be using it? What's an appropriate use of AI? A lot of work there.
One of the teams, and this gets into the second half of your question, one of the teams that reports up to me is our data science team. And I saw earlier, Shikha Mohindra, our principal data scientist joined and she had IM'd me earlier that she was going to join. So at some point, Shikha, I may have to ask you to come off mute. You may have to bail me out on some questions. But with that being said, we stood up a data science team a few years ago. It's a team that economically, you have to be at a certain scale for it to make sense. Data scientists, to do it right, you need a balanced team, domain experts, some more, I would say like statistics, modeling experts, some technology experts. It's really hard to find that one unicorn person or small team that can do it. So you really, to enter that and do it well, you need a critical, massive team, four or five, six people. You can't do it with two. It's probably not going to work. And then that being said, those resources are not the cheapest resources. So you got to have a certain scale to your business for that to make economic sense.
And we crossed over that bridge a few years ago and we got approval to build out a data science team. Shikha was a core person. We sort of built that team around. The data they use is, it's insurance data. So it's pretty much what you would expect. It's data about our network providers, dentist data, where they're practicing, how many patients they're seeing, how long they've been in our network, that type of data.
Then claims data is a big one. Claims effectively, the way we ought to be looking at that in a highly transactional world is claims are effectively a receipt for the care that got rendered. So there's a lot of richness to the data there and then eligibility data. So the provider data tells us who performed the service, the claim is what was done, and then the eligibility tells us who got the service, who was in the chair getting the dental treatments. And there's a lot of different use cases for that. And we have to look at that data differently. At times we're looking at it as cross-sectional data, sometimes it's more longitudinal. We've had some great wins lately with quality of care type models and data work that we've built. We've built some great apps that our teams can use out in the field, powered in the background with a lot of heavy lifting and a lot of hard work, a lot of Python being done in the back on that.
So that rounds out the data. I'm trying to think through what data we wish we had, because it's always great to think about the data you have, but the data we wish we had. We've just recently started bringing in third-party data where we can buy or rent data from third parties. And we're still trying to get our arms around the best uses of that. We're probably not as mature as we need to be, but we're getting much better.
Third-party data and fraud detection
Oh, that was going to be my next question was, what about third-party data? How does that work? And so that's always something that's really interesting to me. And I know when we've talked with other insurance folks in other areas, not health insurance or dental insurance, more property and casualty insurance, that third-party data is such a huge thing. But I don't know what that looks like when you're talking about health and when you're talking about dentistry.
Yeah. So third-party data in PNC is, you're right, it's super valuable. Data from the US flood surveys, that type of thing. And that's obviously going to be a huge thing for PNC actuaries. When we think third-party data in our world, it's honestly more around how can we get a more full picture of provider behavior? And we're really looking for either end of the curve. We have an obligation, a moral obligation to be looking out for fraud, waste, and abuse. So we're looking at that. And that is a tiny, tiny fraction of the dentistry going on out there. The other end of the curve though, what are those dentists that are providing just excellent care, excellent patterns of behavior, conservative dentistry, they're keeping their patients healthy for the long-term. Those are the dentists we want to look at. We want to reward those dentists. And sometimes we're looking at third-party data to do that.
I'll zoom in on one side, on the fraud side. We did get data around financial behaviors. And no surprise, you find a dentist that's got tax liens and has filed for bankruptcy and maybe has borrowed money a bunch in the last couple of months. And yeah, that's a dentist we want to take a look at. On the other side, you've got dentists who have gotten awards clinical. Well, we want to know that too. Those are dentists we want to look at on the other end of the bell curve. So we're looking at both ends.
R, Python, and language choice
Awesome. Well, I have some questions rolling in through Slido. So even though I have some of my own, mine are going to wait just a second. So let's go over to a Slido question. Arsenis asked a question. Arsenis, I'm going to highlight your question in Slido so everybody can see it. And also, if you are able to unmute, you can ask yours. Awesome. Thanks, Libby. Hi. And thank you, Toby, for joining us. My question is actually really simple, something that you just said. So you mentioned that you were introduced to R in your kind of studying actuarial and all that. But your team also uses Python for certain things. So I was wondering if you could discuss maybe how you all decide which language to use and for what applications, and maybe how you use them kind of interact, like going from one to the other.
Yeah, this is a great question. And this is one, even I asked the team the same question. So you're not wrong. I would say the four main languages or tool sets that are used in that area, we do get a lot of data pulled and manipulated through SAS, which is SAS was here when I came here in 2003. I'm not sure if we were building it out today, we would be using SAS, frankly. But it serves a purpose. We're highly transactional in data. SAS, the way it operates is very much process one row at a time. That's the way our data is structured. The paradigm works fine. Then R, Python, and then just like core grassroots SQL kind of work. Those would be the four biggies. And I would argue we are at that point now where it's sort of modeler's choice, whatever you're most comfortable with. Sometimes we have a few hurdles with our IT security team getting things downloaded onto laptops and things. We've got things tightened, buttoned down pretty tight as we should. But in general, it's modeler's choice and whatever they're most comfortable with.
I am seeing a huge shift though in the last five years. Youngsters getting out of college very much on the Python wagon. R is a little bit less common now that I'm out of the Actuarial area. If I put my toe back in the Actuarial area, I think I'd see a lot more R than I do today. Great question, Arsenis. Thank you so much.
AI governance and the "guilty until proven innocent" approach
All right. I saw a question from the chat. I don't normally have the ability to do this, but I saw one from the chat from, let me see, from Runhua who always asks amazing questions. And it's in Slido as an anonymous question, but I was wondering if you'd be able to unmute and ask that. Oh, yeah. I think, you can hear me? Yes, yes. Yeah, for the large model, the general AI is expanding and game changer. And I just wondering how do you make sure your AI pipeline is accurate and ethical and under regulation because you are healthcare data, so that's more like restricted.
Yeah. Yeah. I mean, you're spot on. And the departments of insurance, which regulate us in our states, are fairly skeptical of AI at the moment. Some states, I would say, are outright hostile towards it. That's maybe a bit of an overstatement. But definitely asking questions about AI and your AI governance and how you're thinking about that. The way we've chosen to do it is a little bit of a guilty until proven innocent approach, where you're effectively not allowed to use AI until you've gone through an approval process that says, yes, you can. I think as we learn more about it, we'll probably change that approach.
The way we've chosen to do it is a little bit of a guilty until proven innocent approach, where you're effectively not allowed to use AI until you've gone through an approval process that says, yes, you can.
But as of right now, it's a two stage process. The first approval is we have a use case for AI. We pull together the documentation, and it's pretty quick. It's not a big deal. It goes to a team that thinks about it through an ethics bias governance lens. And they'll ask questions like, what data is being used? Is that data staying within a dedicated tenant? Or is it being used to train a public model? So open versus closed LLM. Does the data ever leave shore? What tool are you going to use? And really what that team is trying, that approval process is really trying to answer two questions is, are we OK and proud of what's happening with the data for this use case? And then should AI be used in this use case? And sometimes we're finding it's really, it's somebody wanting to do something fun and cool, and there's actually a better way to do it that's not AI anyway. So stage one is what tool and what is happening with the data? What's our data posture? Stage two, assuming it passes one, is more where the CTO and the architects get involved. Are we going to have to integrate? Are we going to have to punch a hole in a firewall to get to an external tool? So it's a little bit more around the technical, how would we do it? And do we have the resources to throw at this? And is it worth our time to do it? So it's a two stage process. The assumption is you cannot use AI. And then further down the road, you can prove your case and get the approval to use it.
I love that, Toby. Thank you. I especially love the acknowledgement that sometimes the answer is just like, oh, we could do that with AI, but we could also just not use AI and still do it just fine. Yeah. Oh, yeah. Well, and the funny thing, it's funny you say that, Libby, because I also, I sit on this beast, the IAA, International Actuarial Association. I sit on an AI task force for them. We just had a call last week. And in Europe, and if there's Europeans out here, please chime in in the chat. I'd love to hear your thoughts. AI is being thought of more from the environmental or carbon footprint of AI. And they're looking at it more, every time you send that chat GPT or copilot query, you're basically ripping down a tree somewhere. So be careful when you use it. And they're saying, yeah, we shouldn't always use AI. It's not always something that has to happen. So it has been interesting to hear that, that there is this awareness that AI might work and might be a candidate solution, but it doesn't have to always be the solution.
Yeah. I'm finding often that people are like, oh, here's all these things that AI can do. And that helps them think about use cases, which is sort of a backwards pipeline. But then if you talk to more people who are really experienced in things like NLP, you realize, oh, wait, we can just use NLP to do this and you don't have to use AI and we'll be just fine.
Well, and I think, sorry to jump in there, but the other thing that I think different, yeah, the other thing that to me, at least differentiates AI from other things we've seen, just because of the scale of what it is, is it seems like other things that in my working career have revolutionized things. I think like the adoption of the internet, that was something that technology came along. Then we all figured out how to adopt it and implement it. And it certainly did change things. No question about it. AI, the fundamental pieces of AI have been around forever, since the fifties. We've just never had big enough data sets and enough compute power to do it. So it's like, this is a weird case where the technology has caught up with the science. It used to be the other way around. And so I think we're all struggling with that just a little bit.
Yeah. All right. Great. Fantastic question. Rudhwa, I'm going to look through Slido here because we suddenly have a bunch more Slido questions, which is amazing.
I have an anonymous question here that is a little bit more of a career advice question, if you're up for that. I will highlight it in Slido so everybody knows. My career background is as a data scientist. I have just made the step up to management and I'm feeling a bit overwhelmed. What advice can you give me and how can I know if I'm doing a good job? And you had a pipeline of moving into bigger and scarier roles. Maybe you have a little bit to share here.
Yeah, it is a good question. I love this question too. And I'll go back and give a little bit of background. Not that I want it to be biography or anything, but my bachelor's degree is in statistics and math. And when I got a bachelor's degree, I thought, okay, great, cool. What do you do with a bachelor's degree in stats? And it turns out probably not a ton. If you get lucky and you find a niche, that's great. Master's degree or doctorate for sure, but a bachelor's degree, a little bit limited. So I wasn't really sure what I was going to do. I stuck around, did some research with a professor, took some graduate courses, ended up thinking teaching could be the thing. So I ended up actually went back post-baccalaureate and I taught. I was a high school teacher for a while. I taught high school all day and I taught at a college at night. I thought everybody worked from 7 a.m. to 10 p.m. I thought that was normal. Turns out it's not.
And along the way, I kind of stumbled onto the actuarial career and resigned from the teaching role and went into the actuarial role full tilt and did a little stint in consulting, ended up here at Delta and worked my way up. And throughout there, there were always those challenges. And this comes back full circle to the question. There were always those opportunities. There were always those challenges. Even if it was within the actuarial lane, like taking on a new sub team, or maybe taking accountability for underwriting as well as actuarial. And for me, those leadership roles, I just always, I never said no. I always knew and I trusted that I could figure it out.
If you think you shouldn't say yes to it until you have it figured out, you'll never say yes to it because newsflash, nobody ever has it figured out. The first time you lead teams, you're going to realize that it's never quite what you think it is. In fact, I have to check myself when people come into my office and say, hey, you know, I want to get into management. I have to make sure I take an inquisitive tone. I have to say, oh, really? Why? When in reality, I want to say, why? Like, why would you want to do that? But it is also very incredibly rewarding along the way.
The four steps of managing
I would argue that the biggest thing you can do if you want to succeed in a leadership role is no matter what advice you get, there's an underlying assumption of all of it that you know your team. You can't actually implement any of that advice if you don't actually get to know your team, and you'd be shocked how many managers skip that step, especially if they inherit a new team that's not theirs. They just kind of skip over it.
I tell my folks, and I tell you, I'm pointing to the other building because that's where they're sitting. They're all, and probably Shikha is going to plug her ears here in a minute because she knows I'm going to say it. Managing, I think it's a four-step process, and we overthink it very, very frequently, but it is the same four steps just on rinse and repeat. That's not to say they're easy, but it is the same four. So, I mean, if you take anything from it, I would take this.
Step number one is put the right people in the right role, and that's an ongoing battle. The minute you get your chessboard set up, the game changes or the opponent moves or whatever. You got to move people around. I just this morning moved some people around, and some were happy, some were not, but it had to be done. So, putting the right people in the right role, and then as a leader, you have to give them the right vision of what you're trying to accomplish. Make sure it's really clear, and we're all aligned on what that is and why we're doing it.
Give them the right culture to get it done, and then the biggest one, number four, is get out of their way, and if you've done the first three, you got the right people, you gave them the right vision, you gave them the right culture, you should be delighted to get out of their way and let them do their thing. They will knock it out of the park every time. If you're not willing to just get out of the way, you probably need to look in the mirror because you either didn't do steps one through three correctly, could be true, or your ego won't let you get out of the way, and you're in there micromanaging when you shouldn't be. So, it's the same four steps. Right people in the right role, vision, culture, stay out of the way, and you'll be shocked.
Give them the right culture to get it done, and then the biggest one, number four, is get out of their way, and if you've done the first three, you got the right people, you gave them the right vision, you gave them the right culture, you should be delighted to get out of their way and let them do their thing. They will knock it out of the park every time.
I'm so impressed by Rachel who just put your four steps in the chat. Oh, wow. That was so nice. All right. Thank you, Rachel. There's a fantastic question, and also, like, anonymous asker, what a good question. Thank you so much. I wish you would put your name on this so I could thank you. If this was you, send me or Rachel, like, a DM in the chat.
There is a question from Sarah, which is really right in line with this. It's a great follow-up. She says that she cannot unmute, so I'm going to ask it for her, and I will also stick it in the chat so everybody can see. It's how do you match the team's skills and interests to projects that are coming down the line? So, is leadership receptive to new data science techniques or products, or do they take some persuading? I think that's kind of two different questions. Yeah, I think it is, too. Yeah, it is two different questions, and I'll answer them both.
So, the first one, how do you put the right people in the right role? I think, again, it comes back, you've got to know your team. You've got to know their interests. You've got to know their skills, strengths, weaknesses, and I'm a firm believer in the stretch assignment that you think they can accomplish and pull off. However, you've got to make sure you're not stretching them so much that it's demoralizing, and that really just means you really do have to get to know them and get in there and understand what makes them tick a little bit, and the support along the way is important.
Just one little anecdote. We were kicking off a major project a couple years ago, and our normal cadre of leaders were all way overcommitted, and so we picked somebody who I thought was a year or two or maybe three away from taking that sort of a leadership role, but we had no choice. We had to let her jump in early, and we did. We brought her in a room, and she said, you know, hey, I could do this one of two ways. She laid out two approaches, and she said, what do you think, and the minute I looked at her, and I said, no, no, Maddie, it's not what I think. What do you think? What do you think we ought to be doing here? She immediately was 12 feet tall, and she walked out of the room. She's like, I get it. I get what I own this. It's me. I got to run with this. I said, exactly. She ran. She left the room, and I immediately looked at her boss, though, and I said, do not let her screw this up, or we're going to ruin her forever, so we've got to support her and make sure she succeeds. The one time she sticks her neck out and says, I'm going to own this, and I'm going to run with it, maybe a year or two early. If she fails, then we've ruined her, and we did not. We made sure she was successful.
Oh, good. I was going to say, we kind of did an update. No, no, no. It was awful, so I think the first question is those challenge assignments that put people on a path to a career. I'm a firm believer employees need to own their own career development. You, as their manager, you support them. You get roadblocks out of their way, but they have to own their own career. This is not for you to do for them, and I think that's true the further up they go. If they're entry-level, you got to own a little bit of it, but the further up, when somebody's been promoted once, twice, three times, you should be owning less and less of the career the whole way along.
Communicating data science to leadership
The second question, though, was management being ready for new techniques. You do have to do a little bit of a marketing thing. Being able to communicate the results and explain them is a huge, I would argue that's the lion's share of does it actually succeed and go to a production-ready or not. I used to say it with actuaries. I'll say it with data scientists. Now, I will take a mediocre data scientist with excellent communication skills over an excellent data scientist with mediocre communication skills any day of the week.
Now, I will take a mediocre data scientist with excellent communication skills over an excellent data scientist with mediocre communication skills any day of the week.
It really is an internal marketing thing, especially when we're lobbying for dollars and resources. Sometimes you're talking with departments. That's not the world they live in. They're not in the data all day. They're not up to date on the latest models or vocabulary, so you do have to do a little bit of a marketing campaign with it.
Leaning into challenge assignments
Amazing. Toby, I feel like this has turned into the career advice session that I did not know that I needed. Rachel put a question in the chat that I would love it if everybody could go hop in and talk about, which is what are some tips for leaning into those challenge assignments as the individual contributor? She also asked, how do you identify the challenge assignments? That could be on both sides. How do you know this is the right thing with the stakes that are right to give to somebody? Then how, as an individual contributor, do you say, that's going to be worth stressing myself out for? That's going to be worth the challenge because I really want to do this.
What we've been successful doing, both in our data science area and in the broader tech area, is taking huge initiatives that we all well understand. They're just mammoth kind of things. Rewriting an entire application from scratch, which isn't all going to be a two, three-year project, but being able to break it apart into pieces and smaller bite-sized pieces that you can let somebody stand up and own this small piece. It might only be 10% of the whole project, but it's a critical part and they know it. They understand its part and let them own a small part of it. Now, not every project lends itself to being broken down that way, but don't be shy about spending a day or two with a whiteboard trying to figure out ways to do it because it'll pay off. That's on the giving of the assignment.
On the receiving of the assignment side, I don't know that I've ever said no to a stretch assignment that was thrown my way. I think you can find the lesson to be learned in pretty much all of them, but don't let yourself get in over your head. If you can find a mentor, if you can find somebody in an adjacent area, or if you are doing that 10% of a big project, some of the other people that are owning the other pieces and make sure you're asking for help, you're bouncing ideas off early. I have no problem with a new leader getting in over their head and we have a $10,000 mistake. I'm good with that. I just don't want a $10 million mistake. Call for the help early if you need it.
Awesome. That was my question about stakes was like, what is the right stake level for giving someone something that they need the leeway to make a mistake on? Because that's one of those things. If you're giving a big project that you cannot make a mistake on, failure is not an option. To me, I'm going to say no to that one because I always need that leeway when I'm doing something big and new. All right. That was a wonderful question, Sarah, and wonderful follow-up, Rachel. Thank you so much, both of you.
Sharing data as an API or for research
We had a question in the Slido, which I think we can cover really quickly. It's a bit of an oddball question, but it's never been asked before, so this is interesting. It's, do you share your data as an API or for research? Now, your data is probably full of PII, right? You cannot do that, but do you know of anybody who does? Because businesses and organizations consume third-party data, but we don't ever really talk about whether or not they put data back out there. Yeah, so oddly, anonymized and de-identified data, HIPAA has some fairly strict standards of what qualifies as de-identified. The good news with that is there's tools that'll de-identify it for you.
So we do use a few of those tools, and we have. We've collaborated with research organizations. We've collaborated with some private third parties. We tend to, this is a weird distinction, but if we do that, we make sure we've got all the right governance in place and the right permissions. We tend to do it for free because we don't ever want to be in a position where we've profited by selling the data. Like, that's just a bad look. If we're doing it and there's an oral health research project that's likely going to contribute to the body of knowledge around oral health, that's a worthwhile thing to do. We don't want to be seen as profiting from it, though, for all the right reasons.
The other thing I would say we have to be mindful of is we have some large dental customers that just forbid us from doing it under any circumstances. So we have to pull their data out, which is fine, well understood. The other piece is a fair number of our customers, and probably a lot of the people on this call, are covered by a dental plan where your employer actually still retains all the risk. It's not actually an insurance company that retains the risk. The insurance company processes the claim, maintains the network, but in effect pays it out of your employer's checking account. It's not really how it goes, but effectively. And in those cases, it's not really even our data. We're merely custodians and we possess the data, but we don't own the data. And so in those cases, we need those customers to give us affirmative approval to use it in that way.
That makes sense. What a great question. That was an anonymous one. Yeah. Kudos to you because it's one that has literally never come across.
Yeah. And one thing on that data topic, one thing that makes it interesting for us is we have some customers, one of my favorite customers I work with, we have been their dental insurance plan since 1974. And they have long tenured employees, some multi-generational employees. So there are thousands actually. It's not just like one or two, thousands of people out there that are in their sixties now or fifties that have never had any coverage other than Delta Dental through us for their entire lives. So if you want to ask what happens in a person's mouth from birth to age 60, we actually have a sample of thousands where we can actually credibly talk about that, this idea of continuously eligible members. And that makes the data pretty fun to play.
Amazing. Toby, Isabella had requested Delta Dental's tooth fairy data. We do a study every year and the tooth fairy has gotten outrageously expensive lately. The Delta Dental Plant Association does actually publish a survey every year of what is the going rate for the tooth fairy. And it's more than you think. Hilarious.
Green flags when hiring
All right. Let's move on to a question in Slido from Zach. And Zach has a little asterisk next to his question, which means I will ask it for him. If anybody ever wants to ask a question, but you can't ask it, you always put an asterisk in there, a little note that says, Libby, can you read this? Zach says, when you're looking to add new team members to your team, what are some green flags that stand out beyond the qualification and the job description? Questions are moving around. I'm going to highlight Zach so it doesn't move around for me. So has that changed in recent years? Because the number of applicants has increased and the number of new roles has decreased. And if anybody else has been through the job search in the past couple of years, you know that you go look at a job on LinkedIn and it's been posted for two hours and it's got 500 applicants on it, right? And you're like, oh no. So this is a great question, Zach. It is. It's outstanding. So I would say when we're screening and triaging resumes, particularly in the data science area,
what I push the team, and Shikha's on the line, she can certainly jump into if she needs to, we look for balance in that team. As I mentioned earlier, data science is such a broad range of skill sets and a broad range of domains that are relevant. We've all seen the VED diagram of what makes a good data scientist. That's kind of verbally where I'm going. And you're not going to find very many of those unicorns that live right in that sweet spot of overlap. So you're going to have team members that are maybe a little bit stronger on like the ML ops side of things. You're going to have a team member who's maybe a little bit better with the data engineering, somebody who's a little more domain knowledge, somebody that's a little more modeling and statistics oriented. And it's about balancing the team. So when there's an opening, it's like, okay, what are we missing? What do we need a little bit more horsepower? Let's triage the resumes and go try to find that gap if there is one. So that would be sort of one.
The other piece I would say is kind of what you were just hinting at. The job market is flipped on us where an opening goes out and we're getting hundreds of resumes. I mean, even for a summer internship, we got like triple digits in terms of resumes. And that's a lot to go through. And what I think makes it particularly challenging is there's no barrier to calling yourself a data scientist.
I joke, but I don't think I'm far off where you can have a resume for a position that the person, the candidate, maybe she has a PhD in math and she's done a ton of this work for years and she's super well experienced and she calls herself a data scientist. The next applicant has a sociology degree, which that's a perfectly fine degree, but took one stats course along the way and they're calling themselves a data scientist. And when you dig in, it's like, well, I can open Excel and I can do save and save as. So I'm a data scientist. And that's not exactly what we're after. So vetting through that can be a tough, tough thing.
Hiring for traits over skills
What I've decided lately because of that is the right way to go here is we're big on interviewing for traits, not skills. Skills, as long as you're somewhat technically inclined, you can pick up the skills along the way. I mean, that's one of the beautiful things about Python and to maybe a lesser degree ours, it is pretty easy to pick up if you're technically inclined. And things like RStudio make it easy to do. So I would say we're less cranked up about discrete skills, more traits. And by traits, I mean, are you curious? Do you ask good questions? Are you a lifelong learner? How are you going to work in a team? How do you attack a big problem? Do you try to solve it all at one time and you have that no progress, no progress, then at once it all kind of snaps together. Is that your mode of solving it? And that's fine. Or are you a break it apart into small pieces and solve it? And we need both. We just need to make sure we're getting the right balance. So we spend a lot of time in the interview and screening process trying to get to those traits rather than discrete skills.
What I've decided lately because of that is the right way to go here is we're big on interviewing for traits, not skills.
My other reason, and this applies more on the application development arm of my job, perhaps, is we found when we were really cranked up about skills, like do you have the latest Java certifications, those types of things, by the time a skill is realized in the marketplace, it's adopted by a university curriculum, they pump out a couple of graduating classes, there's some certifications achieved, that's probably five or six years on a date already. So getting all cranked up about a particular certification or skill, I'm probably less excited about.
Yeah, there's a little conversation in the chat about all of the different things that the same job title can be. And even the job description sometimes, you get into the job interview and you're like, oh, what you want isn't even what's in the job description. Yeah. And I love this one. I've interviewed STEM PhDs who call themselves data scientists and can't answer that. That's the other problem is, yeah, they may know the domain, but you ask them how to join two tables or what's wrong with this join. You assume they are kind of deer in the headlights. So it's, yeah, that is a great point too. You can go too far the wrong way.
I think that this is a good time to say what I always say, which is data science is not one thing. If you are not the flavor of data science that Toby needs for his team, you probably are the flavor of data science that somebody else needs for their team, because they do need that deep subject matter expertise. And it matters a little bit less that you know how to do specific technical things. So there is a place for everybody out there. I think it's a, it's a really, really tough thing writing a job description. And it's also a really, really tough thing being a job applicant and reading through a job description.
So it's tough on both sides. And yet, Shikha, if you ever wanted to hop in, Rachel said in the chat, let us know if you have something to say in kind of support or augmenting any of Toby's answers here. Zach, fantastic question. We have some more stuff.
Sorry, I was figuring out how to unmute Shikha real quick. Oh, yeah. There we go. Thanks. There we go. Hey, Shikha.
And that's, you know, like not at the, not just at the onset of when somebody's coming in, even when we're within the team, like just say today morning, I had a conversation with one of my team members, who can, who can go deep in like 10 separate areas? Because again, like you said, Libby, data science is not just one thing, you can go deep into SQL domain, or you could go deep into MLOps and, or machine learning. So he was asking me, hey, where do you want me to focus? And you know, like, like, we've been taught, like, let's throw it back at them, because the goal is the organization or the team goal. So depending on what we want to deliver, what are the trade offs of going deep right now? Or could we do it later? Because at all times, all of us are thinking about one our profile, like, hey, how does my profile get really good and deep and everything.
But I think, as organization leaders, we have to like, hey, but this, this team impact, the data science impact, if we focus in on that and hone in there, we know our answer, and understand what's relevant within all that data science can bring to the table.
So, so yeah, I mean, it's kind of like a constant battle of which area to go deep into, even when you come in. Well, you know, we had a great question from Samita in Lido, which is kind of a follow on to this. So Toby and Chika, I will let you answer this one together, if you have any thoughts. And Samita, you have a little asterisk, I'm gonna ask it for you. Coming from a core statistical background, how much would you expect from a fresh data science graduate about statistical knowledge, if you were an interviewer for a modeling role, any specific concepts that you would like to name? Any specific concepts that you might ask them about in the interview, I think is the question. Yeah, if it's a specifically for a modeling role, like that is the niche that the opening is intended to fill, I would be looking for some pretty, pretty solid statistical knowledge from that person. You know, things like GLM's principal component analysis, I'd be expecting them to know the ins and outs, some of the underlying assumptions, what happens when those assumptions are violated, how that model can go wrong, tree based models, a lot of those types of things would be the core of the interview. If it was a job that was had modeling as a component or was modeling adjacent, I probably would back off on a lot of that. But if it is a core modeling job, that those are some things I'd be asking for some pretty deep understanding of.
All right, awesome. Samita, I hope that that was helpful. Thank you for asking that great question.
Keeping the team current and stretch assignments
Let's see, we have lots and lots of questions and only about 15 minutes left. This hour has gone by so fast. Inger, you had asked a good question. Are you available to unmute and ask that one? Let me see if I can find Inger. There she is. Hi, thank you. I have a quick question. My question is, how do you make sure your team stays current with new data science tools or skills or methods? Yeah, this is a fantastic question. And I am blessed with a team that's very curious, self motivated. This, I get it. It's a great question. It's not something I've had to worry about, though, to be honest. You know, we recruit very curious, intelligent, self motivated folks on the front end. It kind of takes care of itself, to be honest.
That's kind of nice, right? Yeah, I get it. I'm spoiled in that sense. I'm totally spoiled. I think part of that answer might be people need the space in their day to be able to go sharpen their axe as well. So not overloading people with work can sometimes be helpful. Like, if I'm not overloaded with work, then I have the ability to go and investigate new things and learn more. It has to be part of my day. And I would say, too, that there's a way to rope this back into a prior discussion around those stretch assignments, too. If there's a particular skill set where I know that there's something this person needs, they're a little bit lacking in this area. When you see those stretch assignments that would make good use of it, assigning it, marrying it up with the stretch opportunity, here's the issue. And I want you to go solve it using this tool set. Then that kind of isn't inherent. It now becomes part of the job, not in addition to the job.
Building a data science team from scratch
Awesome. Okay. Let's see what we can ask here with our few minutes that we have left. How many of these Slido questions can we fit in here?
Edward has a good one. Edward Leland, if you are available to ask this one. I'm going to highlight it in Slido so everybody can see it. Let's see if Edward's with us.
Hey, Edward. All right. Yeah. Thanks for picking my question. Yeah. I just got one about building a data science team because I'm a member of a newish team. This is just something I've been mulling over in my head. Did you have a ranking of necessary roles that you wanted to hire first, data engineer versus data scientist or analyst, or maybe a dedicated reporting staff? And what tools or platforms did you consider essential from day one?
When we first built the data science team, I knew who I wanted to lead it to get it up off the ground. A lot of conversations with that individual. And the short answer, yes, we did know what skill set we were going for. But we also knew during the interview process, if you saw the it and you know it when you see it when you're doing interviews, and it doesn't exactly align, that you're better off bringing the person in anyway and work with it. If maybe that person exemplified your number three skill that you were looking for instead of number one, but they're a really good person and you think they're going to be a good fit, go ahead and take the risk. Do it anyway. And we've never been burned by that. I was going to say rarely, but I don't think we ever have. That's always been the right move. When you recognize the right talent, even if it's not the perfect person for the role, bring them in anyway.
Part of the luxury that we had in that is it wasn't an existing data science team. We were building it from scratch, trying to define what it was going to be. So if we had to pivot what the vision of that team was because of the team we recruited, that was fine. It wasn't like we were trying to fit into an existing structure. So that was a little bit of added flexibility. Now that the team is up and running, I think we're a little more conscious and a little more aware of where there's gaps or where there's a need to backfill. And I think we're perhaps a little bit more rigidly going after a certain skill set.
Chika, you're far more in the trenches on the hiring than I am.
Okay, you guys can hear me now? Yes. Like Toby said, initially we kind of started off as a blank slate. And I do want to mention that within our culture is like the culture to encourage experimentation. So even if somebody brought in a particular domain experience or a particular experience, we kind of thought about it as this vector diagram that Toby alluded to, that holistically as a team, we've kind of got everything covered. But now as we, you know, as the kind of demand increases, as we're maturing within the organization, we identify another kind of gap. So the new team member, we have to make sure, say statistically, like we have to make sure they know the assumptions of the GLM model or like a particular domain that I know my next two years or three years what my deliverables are. We want to make sure that that person's going to address our gaps. So it started off like as broad skill sets. Now we're getting like, hey, I definitely need this. We've got everything else covered. But there are always, with that said, there will be overlaps. And the core curiosity, like, you know, excited about learning, you know, getting a win for the team or like focused on our mission, those still remain, you know, besides the skill sets.
Thank you, Shikha. All right. Thank you for that amazing question, Edward.
From teaching to managing
Oops, sorry, Shikha. All right. I have another question here that is from a little bit down in the stacks of our Slido questions, and it's anonymous, but I really want to know the answer to this, too. So it is how much of your background in teaching high school students has spilled over into managing your team? And the reason I'm interested in this is because I love teaching. And the reason I'm interested in this is because I love teaching. I steer myself away from management. I'm wondering how those things go together. And also some of the most wonderful leads I know were statistics teachers in high school as well. Yeah, you know, there is a fair amount of overlap, you know, motivating a classroom of freshmen to understand algebra and leading a team to tackle a project. There is some overlap. I'm not going to lie. I think a lot of it is the presentation skills. That's been the big thing. I spend a lot of time in front of a classroom. Now it's more a lot of time in front of boards of directors and customers and things. So I think that certainly is useful, being able to think on your feet. One of the other skills that I think teachers take for granted is they're excellent planners. You're planning out weeks ahead of time and not being rattled when your plan goes off the rails a little bit, which it always will. Being able to pivot and adjust when a plan gets in the weeds. That skill set that I picked up in the classroom definitely has been valuable and useful. And then there's always that, you know, every teacher has a little bit there in them that's a bit of a mentor, a little bit of a coach. Talking with kids about what are they going to do after high school, career kind of related
things. Those skill sets definitely play over into how I handle and deal with my teams. Absolutely.
So who's scarier, a group of teenagers or a boardroom of executives?
Oh, that's a great question. I might say the boardroom. I will be honest, the style I had in the classroom, I always used to joke the three populations I had to deal with, the students were the easy ones. Parents and other teachers could be a bit of a challenge. But the classroom full of students, I was fine. Not a big deal. Hilarious. I feel like I'm terrified of teenagers. They can be so mean. Oh, but they're also teenagers are easy to manipulate too. Very easy. I'm afraid they would think I was not very cool. All right. Well, anonymous asker, whoever you are, that was a wonderful question. Thank you for asking it. I thoroughly enjoyed that.
Hiring for traits and resume advice
All right. I wanted to ask at least one more career advice question. Morgana has one, and then I will ask a general one. So Morgana, if you're available to ask this one, that'd be great. I'll highlight it in Slido. Thank you. Yeah, thanks a lot. Toby, I was wondering if you are looking for traits rather than skills when you're hiring someone. Do you have any advice for how someone could maybe emphasize those traits on their resume?
Oh, yes. Yeah. This is the exact question is if I'm looking for traits, how do you present the traits and make sure they get noticed? So let's go back up. Let's even the cover letter. Cover letter is a good chance, a good opportunity for you to insert those traits that you think would be successful in the role. And those are generally are going to be things like curious, lifelong learner, those types of things. You can get those into your cover letter. I like to think I'm pretty good at detecting an AI-generated cover letter, so be careful on that one. Of course, generate it, but then modify it, make it your own. It certainly can give you a head start. On the resume, what I would say, two things. When you're outlining roles and duties, that's great. That's sort of the minimum. That's what everyone's going to look for. I have this role with these duties. If you can pivot that to, and what did that contribute to the company? I grew enrollment in our product from X to Y in Z months. If you can nail that down, that's even better. That'll get the readers, certainly get HR's attention. It'll get the reader's attention. And then what specifically you contributed to it. And that's where you can weave in some of those traits that you think they may be after. Exactly what was your role in that? Because I can be honest too, during the interview, if you put hard quantitative things, we grew from X to Y, the first thing I'm going to ask is what was your part in that? What part did you play in that role? And I want to hear that it was a team effort, but I also want to hear that there is a piece that you owned that you contributed to in some sort of a unique way. And just putting that straightforward in the resume is probably the best way to do it.
That's very interesting. I feel like I hear different things from recruiters versus people when I do interviews. A lot of times when I see recruiters, they want either a really long resume that has everything that you've ever done, so they can find everything or else something that is just very, very bare bullet points. And it sounds like you need more information about each task. Yeah, I would err on the longer side. Yeah. I wouldn't feel the need to get it down to one page. If you've got two, three pages worthwhile accomplishments, I don't think that's a problem. Thank you so much. I appreciate your advice. Yeah, you bet. Thank you, Morgana. Morgana, you always ask amazing questions. And also, if everybody follows Morgana on LinkedIn lately, she's been posting an update right after the Hangout, which is so amazing and full of little tidbits that she remembered. Yeah. So we'll see what Toby codes make it into Morgana's LinkedIn post.
Career advice
Okay. Let's spend our last five minutes here. I want to ask a general career advice question, Toby. Do you have a piece of career advice that has either really, really helped you or that you try to give to everybody?
Yeah, just a generic piece or can I have a couple? Oh, absolutely. Yeah. So I would say, yeah. Okay. I'll give you three. One I would say is when new challenges come your way, I know early on, you want to be careful not to overload yourself with work because you want to do good quality work. But when challenges come your way, be very careful saying no to challenges. That creates an impression and a brand for yourself that may not be great. So be careful saying no, you know, obviously work with your manager and figure out when that does make sense because it will. I mean, there are times where you've got more than enough work and say no is the right thing. But I would be nervous about that early on.
I would also, number two would be when you build your network, which everyone will tell you to do, it's not a transaction. Make sure you're built. I would rather you build three or four deep lasting relationships than 12 or 15 transactional ones. If you take the, I call it the punch card approach. If I'm new to the company, I have to have lunch with somebody from accounting and then you get a punch in your card. Then I have to meet with somebody in HR. It's going to feel very transactional. The person on the other side of that lunch is going to see it for what it is. So I would say rather than 10 or 15 surface level, I had to get a punch card filled out relationships, three or four deeper ones will probably outperform any day of the week.
I would rather you build three or four deep lasting relationships than 12 or 15 transactional ones. I had to get a punch card filled out relationships, three or four deeper ones will probably outperform any day of the week.
And then the third one, which sounds very cliche, but it's shocking how many people don't do it, is the continuous learning. Just the fact that you're all on this hangout tells me that's not a concern with the folks here. And it doesn't mean you have to spend three hours a day doing anything. 10, 20 minutes a day done consistently over months and months will yield incredible results. It's like compound interest for the brain. So I would say that continuous learning, just because you get the diploma, just because you get the job does not mean that challenge is over.
It's like compound interest for the brain.
Yeah, that's true. I do Tidy Tuesday slash Piety Tuesday every week. Love it. And, um, nothing will force you to learn something new or refresh yourself every single week. Like working on something like I, I get together with friends, we get on zoom and we work on it. And every single week the data is different. And every single week I forced myself to like, refresh on something different or push myself. It's not even about the data set. It's about like, what skill this week I, I was like, all right, we're going to refresh on fuzzy matching. And you know, within an hour refresh on something that I hadn't done since grad school. So highly recommend Tidy Tuesday y'all. Somewhere Hadley Wickham is very proud of you. Very happy to hear that.
Well, Yomi had asked like, how can I join Tidy Tuesday? Tidy Tuesday is whatever you want it to be. The Tidy Tuesday GitHub repo is a repo that's full of hundreds of data sets. A new one is added every week. You do, you can ask access this data in R, Python, Julia, whatever. They're just CSV files. Um, and then the whole thing of this is that it's a social thing. Like go post on blue sky, LinkedIn, whatever, whatever you found. Um, it doesn't have to be a visualization, but often it is this week. I posted a table of results that I got for my fuzzy matching and I posted my fuzzy matching code, my little like, um, function that I had written. So it's whatever you want it to be.
All right. Thank you, Toby. Thanks for the plug Tidy Tuesday. It was so wonderful hanging out with you today. Thank you for all of your wisdom that you shared with us. Everybody, if you would like to save the chat, uh, you can go click the three dots in the top of your chat and click save chat. That will help you keep the resources that we all talked about today. Um, but everybody say thank you for hanging out with us, Toby, and we'll see you next week where we're going to be talking with Keith McNulty at McKinsey. I want to do a shout out to say, if you have some people analytics folks across your team, I think next Thursday's hangout will be especially interesting to them with Keith McNulty as well. Um, and I also want to say thank you to Shikha for jumping in here as well. Yeah. Yay. Shikha. Thank you so much, Toby. Thank you. I hope you have a wonderful day.