image: thumbnail.jpg
Data Science Hangout | Mike Miller, Engine | Adjusting for Stakeholder Tendencies
We dove deep into a conversation towards the end of the discussion regarding surveys (whether internal or to clients):
Responder fatigue is a huge concern and something we deal with on a daily basis.
✔️ Keep it short, keep it focused. Make sure you’re asking people what you want to get out of it. It’s easy to go down a rabbit hole with data you’d like to see. Every time you add a question make sure it’s focused on your main objective.
✔️ Five questions, maybe 10 max. More than that you should be paying your respondents.
✔️ Double-barreled questions are bad - check for these. (questions composed of more than two separate issues or topics, which however can only have one answer.)
✔️ Write your question well. Think about the people answering this. Make sure the question is asking what you want people to respond to and that there’s no way for other people to interpret it differently.
➡️ Resources mentioned as well:
✨ Bryan’s meetup talk on Survey Design: https://lnkd.in/dCK_Ga8x
✨ Mentorship program Maisie shared: https://lnkd.in/gwPyH-Mr"
Where to find more?
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
► Data Science Hangout site: rstudio.com/data-science-hangout
► Add the Data Science Hangout to your calendar: rstd.io/datasciencehangout
Follow Us Here:
Website: https://www.rstudio.com
LinkedIn:https://www.linkedin.com/company/rstudio
Twitter: https://twitter.com/rstudio
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back everyone to the Data Science Hangout. I hope you're having a great week and thank you to Rob who covered for me while I was out in Park City last week snowboarding. And Matthias, I heard there was a great session last week. If you're joining for the first time, welcome. It's great to meet you too. I'm Rachel, I'm the host of the Data Science Hangout.
This is an open space for the whole data science community to connect and chat about data science leadership, questions you're facing, and what's going on in the world of data science. I'm just going to let people in from the waiting room real quick. If you ever want to go back and re-watch or share a session with someone who's missed it, we do have a Data Science Hangout site now off of rstudio.com. I'll share it in the chat in just a moment too. We really want this to be a space where everybody can participate and we can hear from everyone. So there's three ways to ask questions. You can jump in live by just raising your hand on Zoom. You can put questions in the Zoom chat and feel free to just put a little star at the end of your question if you want me to read it out loud if you're like in a coffee shop or something. And then we also have a Slido link that I'll share in just a moment here too, where you can ask questions anonymously as well.
And we really love this dialogue that happens live during the Hangouts too, so I love hearing people jump in live. If you do find yourself super inspired and have spoken a few times, maybe just consider holding back a bit for others to jump in too. So we want to try and make this as inclusive as possible and making room for others is an important aspect of that. So just want to reiterate that we love to hear from everyone, no matter your level of experience or area of work as well. But with all of that, I'm so excited to be joined by my co-host for today, Mike Miller, VP and Data Science Team Leader at Enjin. Mike, I'd love to turn it over to you to just introduce yourself and maybe tell us a little bit about the work that you do.
Yeah, sounds good. Yeah, happy to be here. Thanks for inviting me and getting the opportunity to chat with you all. So like Rachel said, I'm Mike Miller. I'm the VP of our Data Science Group at Enjin Insights. So Enjin Group is kind of our, I guess, our parent company. So we have three really kind of main arms. So EMX, which is our digital like ad server programmatic arm, and then we have an agency group which does kind of typical marketing campaigns and creative development and things like that. And then our Insights Group, which I'm a part of, is kind of our market research and primary research arm.
You know, it's kind of built on, you know, survey research and things like that. So within my department, our Data Science Group, so there's about eight of us that kind of handle the data science and, you know, some data engineering capabilities. Our main role is advanced analytics consulting and methodology of primary research. So a lot of proposal work that we come through, we help, you know, design a methodology, what the, you know, what's going to solve the business need, you know, what techniques do we need to, you know, run and, you know, do regressions or, you know, conjoint work, choice-based modeling, things like that. So what's the right methodology and kind of strategy to solve the business need, you know, obviously has some strategic, you know, perspective and kind of future thinking for a lot of our key accounts, you know, kind of planning out, you know, 2022, what's, you know, they got, you know, their budget, they're trying to do X, Y, and Z, you know, so plan out what's the proper cadence or what's the proper, you know, sequence of research, you know, maybe it's the segmentation that we build on, you know, product optimization and activation. So kind of just strategic partners and thinkers, you know, across, you know, marketing and product development and, you know, customer sat work, CX engagement stuff. So really, you know, full custom research.
We do a lot of, you know, innovation as well as kind of our group is really driven to, you know, drive product development. You know, it's not really necessarily creating new products, but looking at different ways or unique ways to use trying through methods, data integrations, a big, you know, kind of push over the past couple of years of leveraging, you know, different data sources, whether that's, you know, client CRM data, whether that's, you know, online, you know, search for like online behaviors through cookies and IP match and things like that to just, you know, keep enriching our data. You can only ask so much stuff in a survey. So leveraging client data or, you know, third-party, you know, data that's available for sale to just integrate that stuff and drive richer outcomes and richer data and, you know, kind of tell a more colorful story.
So, you know, that's kind of our remit. You know, a lot of our stuff is kind of, you know, data stewardship, you know, what's the proper use of data, what's, you know, waiting and, you know, when to wait, when not to wait, you know, what are the bounds of waiting efficiencies and things like that. You know, obviously with statistics, it's kind of the easiest way to lie. So, you know, maintaining, you know, maintaining quality research and making sure the data is actually telling what it is because it's very easy for, you know, our internal clients and even external clients to say, hey, you know, the data seems to suggest that. Well, it might seem to suggest that, but, you know, it's not necessarily saying that. It's not the question asked. It's not spirit behind the question. So, you know, just to maintain your data quality, maintain your engines, you know, quality and, you know, not misleading our clients to say, hey, you know, you might look stupid. If you want to take that leap, that's fine. Like that's on you, but that's kind of not, you know, our, you know, our recommendation. So, you know, data stewardship is really kind of a core aspect of what our team does and what engines tries to do is, you know, kind of inform, you know, quality data practice, quality research, you know, data quality is a huge thing right now with, you know, teachers and stuff with general survey research. So, you know, maintaining data quality initiatives and things like that.
Using R and RStudio at Enjin
So, I guess I can talk a little bit about, you know, how we use R and RStudio. I think the main, you know, tools that we use within our department are, you know, Excel, SPSS, R and Python. We do a lot of work with Shiny for like online simulators and deliverables and things like that for, you know, interactive deliverables that we can give to our client, you know, it kind of lives on and allows the research to kind of have more self-life, you know, allows our clients to kind of, you know, basically teach them to fish to kind of more self-sufficient, you know, and creating things like that for them to click and push and click through, you know, research and things like that. So, it's been a great tool for us.
Data stewardship explained
Nice. Well, can I ask you when you say data stewardship, could you explain a bit more of what that means or just provide an example of? Yeah, I mean, I think it's just, you know, like leveraging the data in the proper way. You know, there's a lot of times, you know, we tried to use our census values. So, going into a survey, we'll use kind of census targets to make sure the data is representative, you know, whether it's representative census, representative of, you know, a client target market or whatever it is. So, we try to balance that stuff on the front end so we don't have to wait very often.
But a lot of times clients will try to just wait to a story, you know, hey, you know, this maybe, you know, rural is too low and we think we, you know, are getting bad results because there's, you know, rural demographic doesn't answer the question properly or like, hey, can we kind of wait that. So, there's a lot of kind of situations there that clients seem to want to tell a story that maybe the data doesn't suggest. So, obviously, there's ways to tell that story, to manipulate the data a little bit. So, you know, it's our kind of responsibility to say, hey, like, you can wait it to certain bounds or you got to, you know, what's your universe you're trying to wait to. You can't just wait to be waiting. If your data doesn't tell you the story, then that, I mean, that's helpful. You know, a lot of times, you know, with primary research, it's a difficult conversation. You spent, you know, 50, 60, $100,000 on research and they didn't get the answer they want. But, you know, a lot of times that $100,000 that you spent on a research study, you maybe saved you millions of dollars before you launched a product. So, while it didn't give you the answer you want, you're still in the long term probably saving money or, okay, we need to re-optimize, we need to change our product, change pricing to, you know, whatever the recommendation may come out of the research, you know, is helpful. But a lot of times clients don't want that answer. They want, they think they know what the story should be. And it's really kind of, hey, we're to run the survey or the research project for validation. Maybe that validation doesn't happen.
a lot of times that $100,000 that you spent on a research study, you maybe saved you millions of dollars before you launched a product.
So, there's some conversations around, hey, you know, can I, how do I make that story come out? Like, do I drop data? Do I drop these people for one given reason? Maybe it's data quality issues that they think may not have answered the question right. So, it's just, you know, kind of managing those conversations, you know, making sure the data, you know, as long as we design and field and do the data, you know, field the survey properly, you know, there shouldn't be any flaws in the data. And if the data is telling a particular story, you know, there's no real grounds to manipulate it or weight it in a way that you think it should go to try to tell your story. So, you know, and also, you know, some of the findings, even with analysis of, you know, running some analysis and maybe taking leaps with, you know, simple, you know, example of correlations, you know, correlations don't, you know, isn't causality. So, while there's a correlation, doesn't mean that, you know, A is driving B, you know, there's, it's a correlation. So, you know, just those kinds of things and helping a lot of our clients are not statistical people, you know, they're marketers, they're, you know, product developers or things like that. So, you know, they're not strong in statistics. So, it's really kind of teaching them and making sure they understand what the results are telling them and encouraging them to not, you know, step out of those assumptions to kind of run with the analysis that may, you know, the statistical method doesn't really, you know, validate that it doesn't support those claims outside of its, you know, general assumptions.
Enjin's clients and industries
Thanks, Mike. I think it might be helpful for all of us to, like, put us in the, like, your frame of mind, too, to understand a bit about Enjin and, like, the clients that you are working with. Yeah. So, I mean, like I said, we, you know, full custom. So, a lot of our studies, you know, I kind of grew up with, you know, I came from MMRSI, which was then BFIRC, which is now part of Enjin. So, I kind of cut my teeth on CPG. So, we did a lot of, you know, cereals and, you know, snacks and chips and different things in general, you know, product optimization. A lot of it, you know, sometimes it's cost-cutting. Can we reduce, you know, the size of a given item and without people really knowing? So, you know, a lot of new product development, what's, you know, what's the flavor? You know, does it meet, you know, perceptions of flavor? Do people like it? You know, is it a test of control or changing a, you know, a sugar content or changing, you know, some kind of recipe? And do people notice a difference without, you know, alienating? We do a lot of insurance work, you know, home auto health insurance. So, just managing, you know, developing products, you know, how, hey, if we want a deductible at $2,000, what's kind of the take-up rate versus, you know, $5,000? So, some of that product optimization, we do a lot of like customer satisfaction. So, like brand health trackers. So, just kind of keeping the pulse on the brand, a lot of CX experience. So, customer experience work, you know, if you go into a store, was it easy to find what you're looking for? Is customer service good? You know, kind of that stuff that, you know, some urgent care work we just did last week, you know, your experience at urgent care. Did you have a long wait time, you know, where you had, you know, courteous receptionist, you know, was the nurse courteous and helpful? Was the doctor, you know, helpful and courteous? Did you get what you needed? You know, was your insurance covered? So, just general experience that, you know, pick an industry. We've probably touched it.
You know, we do some like durable goods. Yeah. So, like durable goods is kind of one of our main clients, like sweeper and development, things like that. So, a lot of that's new product innovation, you know, prototypes and, you know, new devices to the market. So, you know, a lot of conjoint work. So, if we, you know, have a product line, we add something new, what does that do to the product line? You know, is it incremental or is it cannibalizing certain things and then developing the price pricing strategy across that line of items, you know, potentially even stealing share from specific, you know, competitors or things like that. So, you know, the gamut of what we touch is wide, which, you know, I think is a big appeal for why I enjoy my job is, you know, it's the same techniques, the same kind of solutions, but it's, you know, a ton of different industries. You kind of learn enough to be dangerous in many different things. So, the variety is very appealing to me.
Yeah. That's cool. I've definitely noticed it's interesting when you talk about like products, like asking if people will notice if they have like reduced the size of it. I've definitely started to notice that during COVID. Yeah. I was like, yeah, if you start to see things, you know, change or like the recipe or things are slightly like, yeah, I recommended that because people didn't know.
Yeah. So, it's kind of like, you know, you get the backseat of, you know, product development and, you know, you go out to the market, you're buying these products that you recommended to, you know, reduce the size and keep the price. So, it's kind of interesting.
Hiring challenges
I see there's a few anonymous questions that are coming in right now. And one was, are you feeling the challenges of hiring right now? I am. We actually just, so we had two open positions probably since, well, September, maybe. And we just filled one. We have one open, but we're kind of just passively recruiting for that right now. I'm not really sure exactly the skill set we want. Because like I said, we play a little bit of in the traditional research, you know, statistical market research,
you know, maybe more data science. And then we also have some, you know, data engineering capabilities. So, managing larger databases, we don't have a lot of that. So, we're kind of larger databases, we don't have a lot of that. So, we have probably one guy that's really core to our data engineering team. But we don't really have a ton of that work. We'd love to get, you know, more, get our hands on more, you know, data sets and larger data and, you know, kind of do more typical data engineering tasks. But, you know, it's the chicken or the egg. Like, do we need to hire so we can do more of that? Or do we need to do more of that before we hire someone and justify costs? So, we're kind of figuring out what that second skill set should look like.
But yeah, I mean, we struggled for probably a month or two to even find good candidates. We found really good candidates that, you know, felt that they were in range. We got the offer stage, they had another offer that was completely out of our range. They accepted another offer. So, that happened two or three times. We got some resumes that kind of passed the initial screener through HR. They get on a call with, like, a technical member of our team and couldn't even, you know, answer the simplest question. So, you know, some resumes, like, yeah, these look really good, but they can't talk the talk. You know, we've had to change our kind of initial screening criteria many times.
Like, you know, we got on, you know, an hour interview and we were basically done in 15-20 minutes. So, yeah, hiring has been, you know, a challenge. That's frankly how I kind of got connected with Rachel. You know, I was meeting with Kalau, our RStudio contact, and just kind of mentioned, hey, like, we're really struggling. Like, are you connected with any, you know, our communities or different things that kind of post different jobs? And, you know, I met Rachel through that kind of connection.
But yeah, definitely, definitely struggling to, you know, fill positions, even get reliable, you know, I guess qualified interviews. And then, you know, the good candidates have multiple interviews or multiple offers that, you know, we lose for one reason or another. You know, it could be salary, could be opportunity, could be, you know, the location of our business. I'm in Cincinnati, but we have a New York, Chicago-based office, but we're all kind of working remote now. So, you know, remote, you know, situation is certainly open now with COVID and everything's, you know, kind of more accustomed to that. But yeah, so definitely hiring has been a challenge for various reasons.
Screening and recruiting challenges
I know you mentioned you changed the screening process a little bit, but are there other, like, tips or things you could share with us through that process? Yeah, I mean, not really. I mean, I think our job post has largely stayed the same. You know, I think it was, you know, we just started to add some, like, qualifying questions that our recruiter could kind of ask, but, you know, they weren't too technical, because obviously she's not a data scientist. So, like, if they gave a response, like, would she know exactly what they're talking about? Or, like, so we kind of have basic qualification questions that, you know, we kind of armed her with, like, here's a question, here's a general answer, as long as they don't, like, I have no idea what you're talking about, like, would be a red flag.
So, you know, that has worked well. I mean, we've still got on calls where people, like, yeah, I really know Python, I really know R, and, you know, our kind of technical person on our team would kind of ask the interview, and they're like, oh, I don't even know what a regression is, or I don't know how to do that, or I would just Google it, was, you know, one guy's answer to many questions. Oh, I just Google it, I Google it, which is fine. You know, we certainly Google a lot, but, you know, some of the basic questions of, you know, how do you manage a database, or how do you create a table, how do you run, you know, frequencies of things? Oh, I Google it. Well, we need more of a foundation than that, so.
Online simulators and Shiny Server
Yes, I see, Ethan, you had asked a question in the chat. Would you want to ask that one live? Oh, yes. Hello, everyone. This is my, probably, second time joining. Oh, welcome. Thank you. Yeah. Very nice one. So, incidentally, I work in a very similar company, and we also do consulting and market research and insight. So, online simulators and conjoined work is one of the things we do quite often, and I'm just curious about how do you go about managing and maintaining those simulators, especially, you know, when you have to deal with credentials and lock-in? You know, sometimes I used to having a single lock-in to a platform where they have access to all the insights and dashboard.
Yeah. Yeah. So, typically, we'll do it right through Shiny Server. So, we have a Shiny Server Pro account through RStudio, and we basically have a server locally that we hosted on, and we provide credentials to, obviously, our internal teams and then our clients, you know, maybe three, four people that get log-in credentials. They go to a website, you know, we provide the link and publish the, basically, the entire R, you know, package and the app and everything to that server, and, you know, provide it with a link, and they get access to that.
We've also explored, I don't know that we've actually executed this yet, but we have Daprcy for kind of online reporting and things like that for dashboards and kind of ongoing things. I believe we've explored it. I don't know if we've actually delivered one, but kind of leveraging Daprcy to host some of the Shiny Server stuff as well. So, if they already have a Daprcy portal, you know, to your point, like, they don't want to go to multiple different things. They want to kind of one-stop shop. So, I think we've explored implementation through Daprcy, and we kind of host that link to the simulator and things like that.
So, you know, it's been great. You know, I think, you know, one of the main attractions to an online deliverable is version control. You know, we used to deliver everything with Excel, and you make edits, or you, you know, add things to it, and it's, hey, it's version one, version two, version three, and you start sending emails and different things, and it's just, it's difficult to share within the client's organizations. And, you know, hey, is this the final one, or, like, even, you know, three, four months down the road, like, hey, can you send me that simulator? I'm not sure how the great, you know, the latest and greatest. It's always on a link, and it's the same link. We can publish, you know, edits to it, and it's still the same link. So, it's been really good for version control.
It's always on a link, and it's the same link. We can publish, you know, edits to it, and it's still the same link. So, it's been really good for version control.
We've actually had one of our clients kind of regress. So, they used to do Excel. We went to Shiny. Now, they're coming kind of back to Excel because they want kind of that standalone file, you know, I guess that's one downside to an online deliverable is they don't have, you know, access to that. They can't store, they can't keep it, you know, and kind of attach it to a given project. So, they've kind of come back to Excel, which is fine. I mean, just, it has certain limitations, you know, certain large projects, you know, Excel can't handle. There's, you know, functionalities that we've built within Shiny that, you know, Excel just can't, you know, that, you know, Excel just can't do. But, you know, they're pretty basic, you know, there's not many difficult things that they typically do in their projects. So, it seems to work.
Yeah, I guess one of the main downside with Excel is macros and security. A lot of organizations will sort of block those macro-enabled securities, and also version as well. If you use some of the newer fancy function in Excel, you require other people to also have the most up-to-date version of that. Yeah, there's, you know, certain things, you can't, you know, open it with a Mac or things like that. So, some clients have Macs and, you know, just macros in Excel, there's a lot of, you know, issues with, you know, compatibility, so.
Client data maturity and access
Thank you, and thanks, Ethan, for jumping in live, too. Libby, I see you asked a question in the chat, too, if you want to jump in. Hey, hey. Sure, yeah. So, I was wondering if, as a third party who's coming in and helping clients with things, have you ever hit problems around their data maturity or their security protocols? Like, maybe they want something that their data maturity isn't set to actually help them achieve through you guys.
Frankly, we don't get a lot of client data because a lot of our contacts don't have access to it or can't get it. You know, we would love, you know, we have those conversations and, you know, they're on board, like, yep, yep, you know, but they either, A, talk to finance or talk to IT or whatever, like, nope, we're not giving you that data or, you know, don't have access to it. A lot of times, it's a mess. So, they're like, yeah, we would love to get that, we'll talk with our team, and they can't get it or it's missing so much data that it's not helpful.
So, I mean, I think that's a lot of the struggles we run into, you know, there's a lot of techniques that we've explored, you know, like survival analysis and various loyalty databases with, like, grocery stores and things like that, and we would love to kind of enrich their data. And, you know, we have these longstanding trackers with them, and, you know, a lot of access to that data would be very helpful, you know, loyalty, what's a loyal customer, you know, it's somebody that spends, you know, a certain amount of money of their monthly spend, well, they only have customers, so, like, you know, they don't have lapsed people or, like, they don't track them or, you know, things like that. So, there's just been struggles of, you know, kind of marrying up the technique that, you know, would be super helpful that they just don't have the data for, don't have access to it or aren't willing to share it.
Career journey from analyst to VP
Yeah. So, yeah, I started with at MRSI as an intern. And, you know, so MRSI was a Cincinnati-based, you know, market research firm, fairly small, mom and pop, you know, CPG kind of thing. Started as an intern, was hired on full-time as an analyst, very much in the same role, what kind of our department is now. You know, we kind of wrote, you know, did the, did the upfront consulting work and then wrote the full report. Right now we don't write the full report. We just write kind of the advanced analytics piece, you know, kind of more driven to data and innovation and things like that.
So it's a little bit more, you know, my background is in, is in marketing and entrepreneurship at NKU is where I went. But, you know, I, I'm certainly not a coder, which is kind of interesting that I'm here in a, you know, RStudio, you know, data science hangout. You know, I, I have enough R skills to, to be dangerous. I know how to read and kind of manipulate things, not really drafted on my own. So, you know, a lot of, of my role is, is kind of just, you know, guiding the team, you know, consulting and doing kind of, hey, this is how we should do it. This is, you know, the statistics that, you know, kind of nuts and bolts of it to actually execute it through R. You know, I, I know enough to kind of be dangerous.
But, you know, there's, there's other people on my team that, that really justify and help me through those things in terms of like coding. But yeah, I mean, I just, I kind of, you know, have done, you know, kept taking on, you know, some key accounts and, you know, continued through the ranks. You know, frankly, I was probably got here faster than I expected, you know, I think due to some departures in the company, you know, kind of fell into the role probably, I guess, two years ago, 18 months ago. But obviously, you know, the executive team and everything felt comfortable with me taking kind of the lead of the team and, you know, have, you know, our team is, is pretty tight knit. So I think there's a lot of that, you know, they didn't want to bring in, you know, an outsider to something that doesn't really know, you know, corporate culture and, you know, the team and kind of upset to applecart.
So I mean, I think that played a little bit of role. But obviously, I exemplified the skills that, you know, our executive team felt comfortable with that and kind of making the promotion. But yeah, just, you know, continue to, you know, consult and, you know, I guess, you know, execute and deliver on projects that, you know, I've been given. So yeah, no, I mean, I didn't expect to be in this role, you know, out of college 10 years, you know, on job, you know, running a data science team was certainly not my expectation.
Communicating with leadership
So thanks, Mike, when you say you help the team consult and, and figure out how to work on certain projects, how do you go about communicating that to the leadership team? Yeah, so, you know, there's a lot of visibility, you know, we're, I guess, we're not a small company, but we're not a large company either. So, you know, there's a lot of transparency with, you know, the executive team, you know, my mentor was, was on the, you know, the executive leadership team, you know, was basically reports of the CEO of insights. He then became a manager, he's now kind of stepped back and kind of gone part time. So I've kind of shifted, you know, different roles, my mentor now is the CEO of insights.
So there's just a lot of visibility, a lot of access to upper management. And, you know, frankly, a lot of key accounts. So, you know, there are a lot of key accounts are comfortable coming to a frankly, me and be our team that we continue to do a good job and deliver, you know, client expectations and manage, you know, their, their expectations, as well, and continue to deliver, you know, quality product. And, you know, I guess client service is a lot of kind of what we pride ourselves at engine, you know, we kind of, we're small enough to have one point, you know, single points of contact, our clients really get to know our teams, you know, so they'll see people on meetings, it's the same kind of group of people and have a confidence that, you know, it's going to get job, the job's gonna get done. But, you know, in terms of, you know, how the process comes
in, you know, they really, you know, proposal comes in, you know, the sales team will send it, you know, probably to me, and then I'll distribute it across the team. So a lot of our team is, is not necessarily vertical or industry focused. They certainly gravitate towards certain clients that they know and work with frequently, but more technique based. So like, we'll have segmentation kind of experts, you know, discrete choice experts, you know, maybe CX experts, you know, certain pockets of technique. So, you know, depending on what the proposal calls for, you know, it'll kind of get shuffled to various people on the team. And then they'll consult and say, hey, this is the best way to do it. You know, here's how, you know, how many hours we expected to take. And that's kind of our role, like, we don't really, we're not a sales group, we kind of say, hey, this is how we expect to design it, this is how we would do it, here's the hours it's going to take, they put that in the budget, you know, it's the sales team's job to sell it and close the deal. And then we kind of execute on the back end. So, you know, that's kind of the process, you know, how projects kind of move through the shop.
Nice. Thank you. I know you said, you work closely with your mentor, and when you moved into this role, and I was just curious how, or you mentioned the company isn't that huge, but it's not a small company. So do you have a formal process for mentorship, or how do you go about it? Yeah, so they started like kind of a spark is what they call it, you know, they have a spark talent group, and that probably started two, three years ago, you know, based on, you know, employee surveys and things like that, people kind of got stuck or didn't, you know, feel, you know, training was there, kind of the importance of your career development and things like that. So they kind of set up spark training. So they have like a monthly, you know, training topic that people can sign up for and attend, and, you know, some LinkedIn learnings, and, you know, various things that they have that you can kind of do on demand. They also do like a formal, you know, session with, you know, here's management skills, here's change management, you know, topics, here's, you know, goal setting things, just various kind of career management and career development topics.
And then they've kind of identified, you know, along with that, they've identified like key, I guess, high potential employees that they kind of then assigned to, you know, other management or kind of executive leaders as mentors. So within that, I was, you know, I was picked in the high potential groups, and that kind of HR met with me, like, what are you looking for? What do you, you know, how do you want to grow your career? What do you want out of engine? And then it kind of aligned you to, you know, a given, you know, executive member that that kind of, you know, fit those needs and could, you know, fulfill that. So that's how I kind of got grouped with Rich Catrone was my original mentor. He kind of, he managed broader data operations. So obviously, our team, you know, our script and tab group, budgeting appliance, so just general data operations. So I mentored with him for probably a year, 18 months, you know, then, you know, the departure kind of the vice president of our group left. And I think, frankly, that was, you know, the mentorship with with Rich, you know, could see who I was, how I was going, my career progression, you know, and kind of said, yeah, let's, let's promote Mike and let him run it.
So I think, you know, a lot of those connections, a lot of those, you know, mentorship, and just his guidance and my development over, you know, the year 18 months with him kind of led me to that role. Once he became a manager, they kind of then said, okay, what else? What's the next step? How do you want to go, you know, continue progressing, progressing an engine? And that's how I kind of got lined up with the mentor of Don Simons, our CEO at Insights. So I've been with him, you know, ever since I kind of took the role as vice president.
But yeah, it's just continued exposure to the MX group agency group, you know, kind of what's his vision of engine and how I fit into that and how to drive the team forward and what he wants out of us. So, you know, it's just, it's been great, you know, have him as a, as a friend, as a mentor, obviously as a CEO. So you've got to kind of answer to what he wants and what the vision of the team, but, you know, just have a more personal connection with him and, and, you know, can kind of learn and grow and pay, you know, where, where do I need to improve and kind of have, you know, that, that connection to somebody that, you know, not that I necessarily want to, you know, strive towards, but, you know, can kind of lead me in kind of the leadership and, you know, management and stuff like that.
So, yeah, that's cool that you have that program there. I was curious if other people had similar, like mentorship programs at their company. I know that's something we had like a Slack channel for here at our studio, but I don't think we had like a formal program around it. So curious to hear in the chat too from others. I, I see there's an anonymous question too that went back to Ethan's comment and it was thinking about Ethan's comment, does the need to find individuals who are skilled in both advanced Excel and R slash Python further complicate hiring?
Hiring and language preferences
So I think I was just kind of looking at the chat and I think this might get Ethan your follow-up as well. So within our team, we have kind of pockets of people. So like I said, I, I'm by no means a programmer or coder in R or Python. I know enough to be dangerous, but there are, you know, one person that's really good in Python. That's his kind of go-to. There's a handful of people that are, are good and diligent and proficient in R. So people really, you know, we're kind of program agnostic. However you want to get the job done, get the job done. If you're comfortable in R, you're comfortable in Python, you're comfortable in SPSS or Excel, do that. You know, when we look for hiring, it's kind of one or the other. I think a lot of candidates we're seeing now have Python skills. I think maybe that's more so taught in college. You know, some people have enough, you know, they know R, they know how to use it, but they feel more comfortable in Python.
So yeah, I don't, you know, our, our criteria is not to be proficient in both, you know, generally our entry level positions is just have a general foundation. If you're better in one versus the other, great. That means you can kind of learn the other if we need you to. But we don't really require to have all of those skills. You know, I think if you know one program and language, you can learn the other one. You know, it's just kind of general mindset. Yeah, sure there's nuance and things like that, but most of what we're doing isn't, you know, they're not developing code per se right out of like from scratch. We have a lot of stuff that you're just manipulating and editing and tweaking and, you know, kind of, you know, particularly with our Shiny simulators, they're all very templatized and you just kind of manipulate various pieces, you know, on an ad hoc basis on a, you know, given project.
You know, there's certainly some development, you know, there's a new, you know, quirk to the project or things like that. You know, there could be some development and things like that where you would hopefully have some more proficient, you know, users of R and those skill sets. But yeah, I mean, we generally look for just the general coding language. I mean, I think, you know, when we had the first acquisition, we were looking for R particularly because, you know, we needed more Shiny, you know, skills and more proficiency in Shiny and kind of help, you know, we had a gap there. So that was kind of a main focus. You know, now I said with our open position, we're kind of, you know, we're kind of passively looking, you know, there's not a huge rush to fill that. But yeah, I would probably say it's more of a data engineering role. So maybe more Python based, you know, in Azure and cloud computing and things like that is kind of a, maybe the gap or the lean part of the team right now.
So I guess, and again, like we don't have an immediate need for that. So that's kind of, you know, the passive looking, we're starting actually an ATP associate training program, I think in April or May. So kind of more entry level people that are across the company. So various departments will kind of go through and learn, you know, here's how this department works, that department works and kind of get a general understanding of many different departments. So we're kind of maybe waiting a little bit for that. So that project, or that program kicks up in I think, April or May, that they would kind of maybe be in a couple weeks or so before that and kind of jump into that program fresh. So yeah, that's kind of, I guess, our hiring plan right now.
Associate training program
That associate training program, is that so that's something that came from, like, that came from Spark. So that's been going actually a couple years, which I was actually on kind of development team and kind of the execution of that in the first probably two to three years. And what does our data science department in general need from it? Like, what does that look like? You know, how do we, what basic understanding do people not on our team? Should they have data science? How do we operate? It's really just, these are the things you should be aware of. If you hear, you know, now you have a basic understanding or, hey, I heard that buzzword or whatever it is, loop in the data science team, you know, let the technical experts kind of handle those conversations. So it's really just kind of awareness thing. You know, it's, it's morphed and changed, you know, it's no training programs. Perfect. So, you know, we've created, you know, kind of real projects and things for the associates to kind of learn. And this is how our project works. Here's the insights. Here's what data looks like, you know, because most of them are fresh out of college and, you know, various roles and various backgrounds and where they came from. So it's, you know, here's what a data table looks like. Here's, you know, here's what a regression output looks like. Here's how to pull, you know, findings and things out of, you know, these analysis and then tell a story and create a report and, you know, how to deliver that and, you know, present and give those skills to kind of those people fresh out of college.
It's great that you have that. I know on these conversations, a lot of times we've talked about how like time intensive it is to actually train people and bring people on board. Yeah. So originally it was actually like an eight week program that you came in and you did this program for eight weeks. And then we kind of got feedback from, you know, those various managers or those, those individuals say, Hey, like I hire somebody and then I don't get them on my team for eight weeks. Like that kind of doesn't work. So we've, we've morphed it a little bit of, you know, it's still an eight week program, but basically you get, you know, probably two or three days of training, you know, maybe the mornings are, you know, sessions with trainers and things like that and kind of walk through presentations. There's some homework and just general, you know, you know, projects to work on and, and things like that. You can kind of fit in on various afternoons that we kind of block schedule or block time off of them. But then like the rest of the remaining probably half of the week is them to, to really onboard with their respective team, you know, to, to help carry the load, obviously, if they've been hired, that team needs support. So, you know, they can start to kind of fall into that role and do, you know, client work and things like that. So.
Building comfort presenting to executives
Nice. I see there was another anonymous question that was, how do you build comfort talking to and sharing work with executives? Yeah, I mean, so a lot of our, like our, our projects are set up that everybody has, every project has a primary and secondary analyst. So you're always with somebody. There's all, whether it's your manager, whether it's somebody else that knows that client knows that project, you know, or methodology. But it's really just kind of, you know, learned over the years. It's, you know, I, maybe I'm a silent listener for a while and just kind of hear the flow, hear how things work, hear the presentation style. You know, it's a lot of times it's learning the, you know, the, the tendencies or kind of preferences of our internal salespeople, learning the preferences of given clients, what they want, what they expect, how they, you know, are they technical at all? Do they know, you know, any statistical methods? Do they have no understanding of any of that? So kind of, you can speak technical or you cannot. So it's really learning the tendencies and preferences kind of in a safe zone.
You know, you're kind of a silent listener or somebody else is doing it. Then you get, oh, here's a slide or two for you. Then it's okay. You're running the meeting on your own. Then it's, you know, then you start to kind of be that person with, you know, a shadow behind you that you're kind of, Hey, do this and maybe I'll present it. Or, you know, you just start to kind of, you know, learn and adapt and kind of learn a style and just get that comfort level. I mean, there was many times that, you know, early in my career that, you know, I, you know, traveled to a given client. I was there by myself or I was with somebody and say, Hey, you're going to present this workshop, you know, all day thing. And like, oh shit, like that's super scary. Like, you know, I don't know how I'm going to do with that, but you just kind of get thrown into it and you sink, you know, not necessarily you're going to sink or swim, but you're your manager, that person who kind of puts you in that position knows you're going to succeed. You know, is there to kind of fill in the gap, you get off track or don't know how to answer a question. You know, it's, it's kind of those tendencies. I know how, if you don't know the answer, it's just, I don't know the answer. I'll look it up. I'll get back to you. You don't, I'll make it up. So it's just learning those soft skills, learning those presentation skills.
You know, certainly being in uncomfortable positions. And then, you know, as a manager, you kind of learn that, like you're going to grow with challenges. So, you know, kind of put people in positions that they're going to succeed. It might be uncomfortable for them, but, you know, the confidence in my team or my junior people that they're going to do the right thing or I wouldn't put them in that position. So, you know, it's just, it's the comfort. Once you do it once, it's like, oh, okay, that's not so bad. You know, as I kind of moved into the vice president role, I presented at more conferences and things like that. And it's like, oh, okay, yeah, I can present a conference. I've done that. You know, I know the, I know the, the information. I know the, I know the materials I'm presenting. So just, you know, it's the anxiety or the fear of talking in front of people, but like, you know, the subject matter, just, you know, you got to start doing it. So.
Once you do it once, it's like, oh, okay, that's not so bad. I know the, I know the materials I'm presenting. So just, you know, it's the anxiety or the fear of talking in front of people, but like, you know, the subject matter, just, you know, you got to start doing it. So.
Thank you. That's super helpful. And I think just with working with so many different industries too, you have to know about so many different. Right. I mean, yeah, we had a, like a 3d printer client, you know, came to us and I was like, I don't know anything about 3d printing. And you just start Googling and learn about their business. You ask questions, you know, you know, we're market research. We're not 3d printers. We're not
engine engineers. So it's like, Hey, like, I don't really know your industry. Like, how does this work? Or like, Hey, talk a little bit more about what you're trying to do. You know, we're trying to optimize a product or a printer for them. And it's like, I don't, I don't know how that works. Like I don't know anything about it. So you just kind of start to ask, you know, might be stupid questions, but it's like, I, they can't expect you to know their business. You're trying to help them. They're coming to you for things that they don't know how to do. So, you know, it's that give and take. It's just that comfort level to ask the stupid question. And most of the time, like, yeah, you know, that's a great question. Here's how we do it. So, you know, there's, there's certainly things that industries are like, I don't know anything about that. And you just kind of research it, you know, Google it, you know, there's, you know, ask questions of your client, internal client team that, you know, maybe they've had a longstanding relationship with this client and they kind of know that industry or just ask your clients, you know, they can't expect you to know everything about their business.
Yeah, that's a good point. And it's making me think about like how we all even work with our own internal clients or different departments across our own companies too.
I see there's an interesting chat going on right now as well, Javier and Libby around like the responsibility of consultants as well. Javier, I don't know if you'd be open to jumping in and sharing that comment as well.
Yeah, I mean, my comment was more just to follow up to Libby's question. It was, when I was in consulting, I was pretty much providing a very, very specific, like economic and valuation financial service. We did work with a ton of data though. And there was a lot of modeling involved with it. My job was never to be sort of a, you know, data process consultant. So if some company hired us and 99% of the time data was garbage, it really was not my job to try and fix the problem for them. My job was to sort of clean up the data as best as possible, augment the data with, you know, overrides or whatever it might be to get it to a place where we felt comfortable, you know, getting to an end result that was reasonable. So, you know, I don't know how many other data consultants nowadays are in the same situation where it's like, you're providing a specific service, maybe a specific statistical study. Maybe the end result is, you know, a live Shiny app that you're going to deploy for some customer. And if that's the end result, maybe, you know, you shouldn't really worry too much about the process of how the client collects or tracks their data. And, you know, just focus more on how can we clean up their data that needs to be used for the analysis that you're doing?
Yeah. Yeah. No, I mean, I would completely agree with that. You know, I think we consult sometimes, like, you know, we've done some segmentations that, you know, was built on primary survey data, but they also want to tag their database. So like if we, you know, maybe their database has age, gender, some other, you know, psychographics and things like that of their general business consumer, you know, maybe we, you know, can help them tag their database. But, you know, again, like they may have that data, they may have sparse data, but it's like, hey, you know, this is how we can deal with it. You know, maybe you can add some fields to your Salesforce data or something like that, to kind of continue to collect and help you kind of moving forward. But yeah, I mean, I don't think many times that we speak, hey, this is how you should be doing it. Or, you know, generally kind of what you said, Javier is, you know, maybe we're using 10 variables, maybe it has some missing data. Sure. We can impute some data or some percentage of data, and we'll kind of do the best we can with the data we need and, you know, deliver that analysis. You deliver that analysis, and then, you know, the rest of their data problems are kind of their problems. You know, we can kind of help consult and say, hey, it would be great if you could add these data fields going forward. It might help your business, it might help, you know, lead some insights based on what analysis we've done that these are key predictors. But yeah, I mean, we don't get a ton into this is how you fix your data issues.
Mike, just curious, what's the typical duration of your engagements?
I would say a typical project is probably six to eight weeks, you know, for a general ad hoc. You know, a lot of our more key accounts or things like that, it's kind of just an ongoing thing, you know, it's kind of strategic consulting or, you know, planning, you know, it's general conversations, you know, a lot of clients, we have standing meetings of, you know, weekly or bi-weekly of just, hey, what's going on? Yeah, hey, we've got this project, you know, hey, I've gotten, you know, okay, that one's going, it's up and running, you know, kind of getting to the end, hey, we got these other projects in mind, or we're trying to, you know, plan, you know, budgets, and what should we do? You know, what's the sequence of different projects? But yeah, generally, it's six to eight weeks, you know, we have some ongoing trackers that, you know, are fielded weekly, monthly, quarterly. So I mean, those just are continuing general cadence of reporting. There's certainly longer ones, you know, maybe segmentation runs a little bit longer. Because there's a lot of, you know, iteration through that and workshopping with the client. So hey, we get here's the first draft, here's what we're seeing, this is what we think, you know, does that kind of jive, or maybe there's multiple solutions, we kind of work through and say, hey, you know, this is where we think there's here, probably our recommendation, but here's a couple other options. Because again, we, you know, they know their business, you know, we try to learn as much as we can about their business. But there's things internally, there's internal champions, you know, CEO might want x, y, and z that we're just not privy to. So, you know, some of the more advanced are kind of larger projects, maybe those are 10, 12 weeks.
You know, it really kind of depends on you, I think the biggest, the biggest kind of time thing is developing the questionnaire, we just did a major global segmentation across 33 countries, it took us nine months to get the project to get the survey fielded, you know, it was across, you know, 33 plus, you know, every country probably had its own language, some countries had multiple languages. So a lot of translation and things like that, approvals from legal departments and stuff like that. So I mean, it took us nine months to get the project up and going. And then we, you know, collected, collected data for probably two ish months across all the countries. And then we delivered the analysis in about a month. So, you know, it really just kind of depends on on the scope. And a lot of times it's client approvals, a lot of times our clients on the main hang up, you know, to get approvals, get you know, things through, you know, their stakeholders, make sure everybody's got their fingerprint on it, and they feel comfortable with it, if they're spending, you know, a ton of money on it. So a lot of times it's client, you know, client hang ups, you know, kind of sell out a project.
Declining engagements and managing scope
I think part of your answer here answered Robert's question, but I see from the chat, Robert also asked, is it smart to ever decline the engagement or lower expectations if the industry is in a sweet spot relative to your expertise? Yeah, I mean, I think I'm probably a couple years ago, we, frankly, were fairly lean in sales. So we would kind of do the best we could to bid on it. You know, I think now we're certainly in a better spot. But you know, we just somewhat did that a couple weeks ago, you know, we just didn't feel that we could do it on the timeline with the scope and the project, you know, we kind of put our best foot forward and said, Hey, here's what we can deliver, you know, maybe we deliver on one of their three primary objectives, but, you know, just transparent, you know, Hey, this is what we can do. It's gonna solve X, it's not going to do, you know, A, B, and C. And, you know, ultimately, we didn't get awarded the project, but we kind of knew that we frankly didn't really want the project. But I mean, it's one of our long standing clients, you know, they come to us probably two, three projects to propose on a month. And this one, we just didn't feel comfortable with what they were doing what they're, you know, we couldn't, we couldn't address everything in one study, you know, effectively, and at the price point, and you're just kind of just we're transparent, say, Hey, this is what we can do with with this budget, here's what we can do, you know, with these objectives, you know, this is what you're gonna get, you're not gonna get that. And if, you know, as long as they're transparent, I know what they're gonna get, you know, kind of protects us on the back end. But yeah, I think there's been other cases that we've not even submitted a proposal for things or just declined. You know, because, you know, there's a lot of good faith in that, you know, we don't want to, you know, long standing client, we're not going to say, Yeah, we can do it. And then just fail and deliver, it's just gonna hurt the relationship in general.
So thanks, Mike. I see, Brian, you just asked a question in the chat, too, if you want to jump in.
Machine learning and survey design
Yeah, sure, Mike. So a few years back, I worked in marketing research also. And how much machine learning are you doing with the survey data, I was actually able to create some really interesting ways to do it. That were extremely valuable. I'm just wondering if you're finding that are you able to do it? Or is this something that you want to try to convince your clients to accept or do? Yeah, I mean, it kind of depends. I mean, we've done so we definitely do some machine learning on a fairly regular basis with like online behavior, third party data and merging that with with our survey data. So we do, you know, kind of an ensemble approach, there's like five different algorithms, random forces, SVM, XGBoost, things like that, we kind of pick the best model of what we're doing. We've done, you know, we've done some Bayesian networks, it's kind of it's kind of weird, like the research industry goes in, you know, cycles, like things will be hot, like segmentation, we're doing a ton of segmentation right now. And I think a lot of that's from COVID, like just the markets are changing, people are buying things differently, they're interacting with brands differently. So a lot of brands are trying to understand how the market has shifted. So, you know, segmentations are super hot right now. You know, Bayesian networks were probably hot two years ago, like we did four or five Bayesian networks a year. Now we haven't touched one in two years. So, you know, things just go in cycles.
Most of our data from survey research is probably 2000 completes or under. And then a lot of our clients just want to cut and slice data. So, you know, some techniques are useful, some are not just with, you know, the amount of data we have. I think we're trying to engage with our EMX team, you know, in terms of programmatic, trying to get our hands on, you know, different agency clients and try to just onboard the alternative data sources, you know, as we get larger and larger data, that's where kind of the machine learning and things like that. But, you know, a lot of our bread and butter stuff is segmentation, you know, through K-means or even like Sawtooth, they're like CCEA, so it's like an ensemble approach of various techniques or, you know, basic regressions, you know, conjoint work, max diff work, some pricing stuff with like Van West and, you know, price laddering through Gaber Granger. So that's, I mean, that's most of our bread and butter. You know, we're trying to go, you know, be more strategic and try to get more data sources and things like that, which is where kind of predictive modeling comes in, you know, I think is frankly where the future, I think a lot of our clients, they want to know what's going to happen in the future rather than what has happened in the past, which is kind of typical market research, you know, is kind of, you know, backwards looking, you know, what happened in the past, you know, a lot of clients now are like, how can I use the data from trackers or other stuff to predict what's going to happen or, you know, the early indicators of things.
So, yeah, I found one thing was to change the whole survey technique. So you can actually do a 2000 in the survey world is a lot of responses. So are you using a Likert scale? Are you using something else? Yeah, get rid of the Likert scale, go to a zero to 10 scale or a one to 10 scale. And then you leave binary outcomes and you will find logistic regression or elastic net will do amazing things for you. Like a simple way to think about is NPS, right? You ask the NPS question, but then you ask 10 follow-ups or nine or eight or whatever your favorite number is, scale those one to 10. Now you'd be able to say, hey, for every increase in this area that you improve, the NPS is going to go up or down by Y.
Now you'd be able to say, hey, for every increase in this area that you improve, the NPS is going to go up or down by Y.
And what it also identifies is where the problems are. So we, I've actually used that for several companies and it's like, all right, you got to fix this problem. They actually did it. They would spend months doing it. Next thing you know, you do the NPS score again, three more, it goes up three and a half points. And it's kind of like, hmm, interesting. It works. And so it's like, all right, take care of the next question. But the biggest thing is getting researchers to get rid of Likert scale. Statistically, it has too many problems with it. And you'll get nice variability out of a one to 10 scale. And next thing you know, I think you'll find yourself with some really, like even I've done it with 1200 responses and you get statistically significant results.
Yeah. So I mean, that's, you know, a lot of times it's client preference. I mean, we have surveys that have five point scale, seven point scale, 10 point scale, 11 point scale. We just had a conversation last week with a survey, you know, somebody new to their company came on, you know, as most of their survey was 10 point scales. They want to switch everything to 11 point scales, but maintain trending. Like, what does that do? It's like, is there really a benefit to that? Like you're just changing 11 or 10 to 11 for personal preference and you still want to maintain trending. But that was an interesting conversation. But yeah, I mean, we actually have a simulator that's like our CX ROI, that it's all 10 point scale based questions, logistic regression, you know, for every point, you know, on a independent variable, you gain x on the DV. And then we've actually tied that to some of their sales data. So, you know, for every new loyal customer you get, you're gaining, you know, say $500. And we kind of extrapolate that out per a thousand customers and kind of get an ROI feed. So that's actually one of our Shiny simulators that we've delivered to one of our clients. So yeah.
Sounds like one of your clients was a fan of spinal tap or something like that. So they wanted to go to an 11, you know, because it was better than 10. Yeah. Well, I mean, I guess the traditional MPS is on 11, but it's like, you know, okay, but you do want to go to 11, just because you want to go to 11,
or you want to maintain trending that you've done for two, three years. So, you know, and they reported means and they report full scale. So it's like, you know, there's got to be variability, like we can do our best to kind of calibrate it, but there's got to be change. So.
With this like survey topic, I am always curious about like what happens when I just get very tired of a survey halfway through, like I have really good intentions of answering it for the company. And I like the person was great. And I want like their five stars to get through, but then they just keep going. Yeah. No, I mean, I mean, obviously being in this industry, I'm the worst survey taker. Like I hate taking surveys and like, I just pick them apart. And yeah, I mean, that's, that's a huge problem. You know, people, clients, particularly like, you know, hey, let's do this 30 minute survey. I'm like, nobody's going to take it. Like you're going to get 15 minutes in and people bail. So, you know, respond to engagement, respond to fatigue. Like those are huge concerns and things that we deal with on a daily basis.
respond to engagement, respond to fatigue. Like those are huge concerns and things that we deal with on a daily basis.
You know, if you get garbage data, you're going to have garbage analysis. So, you know, make sure you design it right. Make sure you build a questionnaire that asks questions that you want and, you know, cut out the fluff, cut to the point, keep it focused. Yeah. That's my, that's my, was my idea is like five questions, maybe 10 max. One of them should be a binary. More than that, you better be paying your respondents. And when you pay your respondents, you get a whole different, you can get and get a whole different story. But you're right. I mean, survey fatigue, huge issue. Like somebody wanted to do one with like 20 questions. It's like, you're wasting your time. You're not going to get anybody. You'll get like two people to finish it. And then like, or as Rachel was pointing out, you got to worry about, we used to see a straight liners. You get the same score all the way across all five, 10, 15 questions.
Like, it's a difficult thing is like, sometimes that's logical. I mean, sometimes that was one of the discussions. Like I said, I'm not going to remove those just because you think you don't, unless they tell you that somewhere in the survey that. Right. I mean, that's a lot of our data quality initiatives. Like there's, you know, four or five flags and you got to fail two or three of them because, you know, I, you know, it took me 10 minutes to go through the survey, but I straight lined everything. Okay. Maybe that's a logical answer. You know, if somebody did a straight liner in 30 seconds or a minute, sure. They're probably cheating or it's a bot or other things, but you know, it's, it's very, and that's kind of back to data stewardship. People are like, oh, let's throw these people out. Well, there's, there's indication that that's a logical response. So we're not just going to chuck them out because we don't like them.
Survey design tips
I can shoot you a presentation after this that we gave to the marketing group. Share it in the chat, Brian. I don't know. Can you send a file? Oh, I can put it, I'll put it on the meetup. I'll put it in the LinkedIn.
Um, I was just curious to like, with this conversation about surveys, if you had any, just like immediate tips for us all, even if we're like pulling our own colleagues and asking them about like a shiny app that we made or something, are there tips that you have for surveys we do internally? Um, no, I mean, I think, you know, respond to fatigue is the biggest one, you know, ask, ask questions. You know, I think double barrel questions are bad. Like just a general, you know, don't make sure your question is, is, is asking what you want people to respond to. Make sure there's no way for other people to interpret it differently. Um, so, you know, a good question is, you know, write the good, you know, write the question well. So you're getting a response that people understand what you're asking. Um, and just keep it short, you know, keep it focused, make sure you're, you're asking people what you want to get out of it. You know, it's very easy to start going down the rabbit hole of, Oh, I'd love to have this data. I'd love to have that data. You know, every time you add a question that's not focused on your main objective, it's, it's just fluff. People are like, why are you asking me this? Why? You know, it's a five minute question. Well, now it's a 10 minute question because you added five, 10 other questions that don't really pertain to what you want to get out of it.
So garbage in, garbage out, you know, I think that's keep it focused. People are so consumed with a ton of other things. You know, if you can ask a, you know, a five minute questionnaire and five minute survey, you'll get in, get out. Um, so. Thank you. I just want to check that I didn't miss any questions earlier from the chat or from Slido. If anybody, anyone wants to jump in. Give people a few more seconds here.
SPSS usage
Well, I think there was one question about, uh, on Slido, SPSS. Oh, okay. I haven't seen that. I haven't seen that tool in a while. Oh, what? Oh, I see it. What do you use SPSS for was the question. Yeah. So, um, basically every project that we run has an SPSS tool. So, um, Yeah. So, um, basically every project that we run has an SPSS file. So a lot of our, like data tabulations. So we run things through dim net, which is our main, like survey platform that we, you know, script and basically serves a link, um, to have people complete. Um, so that's all run through dimensions and then that kind of exports into SPSS. So when our data tabulation team creates Excel tabs and everything like that basically creates an SPSS folder or an SPSS file. Um, so, you know, if there's, you know, correlations or regressions or things like that, it's just, it's very easy to just hop in there and run it. SPSS, you know, in my opinion has really good kind of that diagnostics. I think a lot of R and Python things, um, you know, sometimes it just runs and gives you minimal outputs, um, you know, in some of the packages. So, you know, there's just a lot of, you know, I think tried and true NIST, you know, that's kind of what I learned in college and kind of grew up learning and, you know, feel comfortable with, and we still use it because it's easy accessible. We have the licenses, um, but a lot of, you know, it's just as easy to export as a CSV and throw it in R or Python if we need it to.
So, um, you know, it's got data labels, it's got the value labels right there. You know, when we get the file, you just open it and see, you know, run basic frequencies, correlations and things pretty quickly and easily. So, um, if people do have follow-up, oh, there's some echo. I don't know where it's coming from. Um, if people do have follow-up questions though, or want to connect with you, is LinkedIn the best place? Yep. LinkedIn is great. Um, you know, that easy way to connect and chat and, you know, can follow up with any questions or anything like that. Perfect. Well, thank you so much, Mike, for chatting with us and sharing all your experience. Really appreciate it. Thank you all for joining today too.