Transcript#
This transcript was generated automatically and may contain errors.
So welcome to the Data Science Lab, everybody. My name is Libby. I'm a data community manager here at Posit, and I am joined by my co-host, Isabella Velazquez. Isabella, please say hello.
Hi, everyone. Thanks for joining us.
We are so excited to be joined today by Edgar Ruiz. Edgar, he is a maintainer of the mall package, which we will be talking about today. Edgar, would you like to say hello?
Hi, everyone. Thank you for joining. Also, thank you for being here in such cold weather, and hopefully we can show some cool stuff that y'all can use.
Introducing the mall package
If you have not used the mall package, and we are hopefully sticking links to everything in the Discord chat. We are talking about a package that allows you to use ellmer to connect to a variety of LLMs and then apply LLMs to your data in a programmatic way, which means like, hey, I have this column of text maybe, and I want to transform it somehow or classify it, translate it, whatever that is with an LLM, and then have a column that represents the output from the LLM.
This is amazing for me, because when I first started using LLMs, I was like, okay, but how do I use this, right? To me, it was just like this is a chat thing. How do I use this programmatically if I want to apply it to my data, and the mall package abstracted away all the difficulty for me, and I think that that's great.
So, I'm going to go ahead and share my screen and just walk through a Quarto doc. I have dark mode or semi-dark mode on my Positron right now. If this is hard for anybody to see, can you please let me know, or if it's too small for you to see, please, please, please let me know.
Yeah, so this is an introduction of an idea that occurred to me when I was looking at some output from LLMs some time back, and I saw that being able to ask it, is this text positive or not, was actually pretty straightforward. Even a locally installed LLM would actually do a pretty decent job at it, and that's kind of what got it started because at the time that I started writing mall, because of how fast everything has evolved, year and a half, year ago, companies were reticent about sending data into the cloud, if it was a cloud provider for the LLM. Now, that's not as bad. More companies are doing that.
So at the time, most companies wanted to keep it local, and it looked like something such as a LLM model was actually going to work well that could be installed locally. So I figured I'd start doing that. So one thing that I can ask folks here today is if you start seeing things that we could actually do better or add, I kind of talked through several different things like the sentiment, classify, extract. Also, you can do your own custom one, and Libby will explain more about what that does. If you see other ones that we could possibly add, that'd be great.
Also, the code is open, and it really boils down to a simple prompt that I'm using to run recursively over your dataset. So improvements and prompts and things like that are always welcome. The GitHub, the link to the GitHub repo is in the website, so please feel free to reach out.
Also, the code is open, and it really boils down to a simple prompt that I'm using to run recursively over your dataset. So improvements and prompts and things like that are always welcome.
All right, perfect. So what Edgar was mentioning, sentiment analysis, text summarizing, classification of text, extraction of text, translation, and binary verification, true, false. These are all built into mall, and then there's also a custom. So what I'm going to do is run through some code where I show you what it looks like to get this connected to ellmer, to an actual LLM, and then what it looks like to use some of these on your data.
If you click each of these, it will take you down to examples of them, right? So we have sentiment analysis, which happens with a function called LLM sentiment. For each of these, there is... for each of these functions, there is a vectorized version as well. These functions are going to be applied to a column in a data frame, but if you wanted to apply them to just a vector, like, hey, I have a vector named this, and it contains a vector of text, then you can use the vectorized version.
So for each of these, there's also, you'll notice, a Python tab up here. That's because we can work on the data frame level. This would be a data frame called reviews, and then we can use the LLM.sentiment function on our column called review. So if you are wondering how things work in Python as you're watching this, please don't fret. Edgar's going to share the Python code after I share the R code, okay?
Okay, I'm going to head back over to Discord, make sure I'm not missing anything. Nora says she needs to learn NLP. Yeah, learn NLP. I love NLP. NLP, by the way, that was my soapbox, still has a place. Big time.
Can I say something real quick about the NLP thing? That's the other thing that I noticed that the local LLM was actually doing real well, even though it may take a bit longer to recursively go through a dataset. It's kind of like you're paying for the time that it will take you to develop your own NLP. Yeah, it's a tax, right? It takes longer, yes, to run, but you didn't have to spend the time doing the NLP yourself and tokenizing your words, and then doing all this other analysis you have to do. So, that's the other reason why this exists. So, as you see the performance, of course, it's going to be slower, but let me see, it's a tax, right? Yeah.
It's also not, I think, as verifiable. NLP, I feel like, is much more predictable. In the chat, someone asked, what's the difference between using mall for these NLP tasks versus traditional or classical NLP? The difference is really just that you are using an LLM versus using traditional NLP methods, which might be like named entity recognition or sentiment analysis using a lexicon, a defined lexicon, or let's say you might be doing something that I love to do in NLP, which is sort of like a power analysis. That means that you are creating this lexicon or this database of maybe words or phrases that indicate certain things, and then you are basically using code and math to compare the text that you are working on to what you have in your database of what is a positive word or a negative word or a neutral word or a powerful word or, you know, NLP has its own topic, but we'll head into our live coding now because we're 14 minutes in and we better do that or we're going to get roasted.
Live coding setup
Please light a candle in the chat for me because I am about to live code with LLMs on the internet and we know that something could go wrong. If it does go wrong, please give me some grace and have patience. So what I will say at the top of the script is if you have not already installed mall and ellmer, those are the two that you are going to really, really need. I have a little install.packages for you if you would like, and I also have this repo for you.
Okay. So I'm going to go ahead and run this chunk, which is just going to load ellmer and mall and tidyverse, and then if you have access to a commercial LLM, you need to make sure that your API key is in your .environ file or your .env file so that ellmer can recognize it and use it. I am going to be using Anthropic Model today, but it's not through an API key in the same way that you would use it. I'm using it through AWS Bedrock. So what I'm going to also show you is Ollama, which is so much more accessible. It's a local LLM that you install on your machine. You can go to ollama.com slash download and I will stick that in the chat.
And Corey had just said, can mall be used with a local LLM? Yep, absolutely. So you can use it with Ollama and what I'm going to do is just show you that I already have Ollama installed and it's running. I don't know if you can see the very top of my screen up here, but on my Mac menu bar, there is a little symbol that looks like Jar Jar Binks to me, but it's definitely, definitely Ollama. And here it is. And I just asked it a question. I said, tell me a joke. And it said, here's one. What do you call a fake noodle and impasta?
So Ollama is up and running on my system. And down here, you can see there's this little model selection for Ollama. I have 3.2 installed. I will show you in a minute how I got that installed and pulled down and like have my system recognize it. But just know that you can choose from a bunch of other models. I'm going to let Edgar hop in here and just mention the size caveat on some of these, because with local LLMs, you're pulling them down onto your machine to run them. And some of these might be quite large.
Yes, exactly. That's exactly the point. The issue is that these have to run in memory. So you may have a hard drive that's big enough, but you still need enough RAM to run these things. LLAMA 3.2 is wonderful because it's very generalized. Going back to the NLP thing, you're kind of limited on the amount of training data you'll have to do your own NLP. But with LLAMA, you have so many more billions of tokens you'll have on text. But the trade-off is that it takes a lot of space. So with LLAMA 3.2, the latest is two gigabytes. The latest LLAMA, which is four, is actually 67 gigabytes on the smaller side. So that's the thing. Unless you have a very nice machine, it's probably going to be one of those smaller ones that you'll need.
Yeah, 3.2 is what I recommend. Also, as time goes by, more folks publish models that are more specialized. You may want to use this in a very specific field. There may be some that are like that. So definitely follow the LLAMA.com website and you can search for your own type of model that you may want to use. But at this point, LLAMA, to me, is the better one. If you all have used another one, then I would like to recommend. Also, please do. Yeah, let us know.
So I'm going to close this and talk about the OLLAMA-R package really quickly. So the OLLAMA-R package allows you to pull down Ollama models and have them recognized on your machine very quickly. For example, I was able to run OLLAMA-R pull and just give it LLAMA 3.2 as my model. And then once that's done, and I'm not going to run it because it already is, I can run this line of code right here that's asking to test the connection to Ollama. And I've got my status 200, which means all good. And it's also telling me that the body is that it's in memory and it's 17 bytes.
Setting up data and ellmer
Okay, so let's head over to do something very important here, which is something I actually forgot to do the first time I did this. But it is now in the repo for you to set a seed so that these random samples are reproducible. What I'm doing here is collecting a few data sets that are full of tweets. This is from one of the Kaggle data sets that's tweets. It's linked in the readme of the repo. So if you would like to go download that, download that tweets.csv. It is also in the repo as data right here.
But if you want to play with it on your own, that's how you can do it. And what I've done is I'm taking a sample. So I have a tweet sample that's just 25 random samples of English language tweets. Then I have a random sampling of Jimmy Fallon tweets. I like Jimmy Fallon tweets because they're sort of a mix up of really neutral like announcement tweets that are like announcing a guest on a show or something. And then also personal tweets from him. And then I have a random sampling of Katy Perry tweets because I'm going to talk about safety and curse words. And Katy Perry sure does like to curse on Twitter. Or at least when this data set was taken.
What I'm going to do next is I'm going to set up ellmer. This is not mall yet. This is still ellmer. It's very important to realize that ellmer and mall go together. You don't use mall without ellmer's support. And it's going to give you the ability to access those models that you are then going to use programmatically. I'm going to do something that ellmer wasn't really built to do. Sorry, Edgar. I'm like slightly breaking it. I'm going to create two different chat objects. I'm going to use ellmer chat AWS Bedrock to set up a chat with Anthropic so I can run this. And you can see over here it created a value, right? This value is just a chat object. And then I'm going to create chat Ollama. And I'm going to tell it the
model is 3.2. So when I run both of those, I now have a chat object. And then I can use a mall function to switch between those different ellmer chats. So ellmer was really meant to, I think, use one chat per session. And so you'll see later on when I call it, I do something special to try to avoid like a caching issue.
But this right here is ellmer's providers. These are all of the different providers that you can use with ellmer to power your mall work. It's a pretty big list. You can go straight to chat Anthropic without going through AWS bedrock, obviously, because if you have an API key, that'll get you there. A lot of people have been asking about Azure, opening through Azure. It's there. Support is there. Cloudflare, DeepSeek, all the things. Here's our chat Olama that we used. If you would like to figure out how to set these up, you could go click it. And then it's going to walk you through this. And actually, this is really, really easy. I think the first time I ran through these instructions, it took me about four minutes to set up Olama and have it running. So highly recommend starting with Olama.
I think, Edgar, you said Gemini is also an option for something that is a more like commercial style model, but that's also a free option. Yeah, that's right. As long as you have API access and a lot of companies are going Gemini right now. So that may be one to try.
I was going to say something real quick, if I can. I just want to like right now, we're talking a lot about ellmer and not mall. ellmer, I went ahead and created the backend for ellmer inside mall because initially it didn't have it. It was only Olama. So that is easier for you to be able to connect to different kinds of providers, right? That way mall didn't have to itself have a code to be able to connect to the different ones. ellmer is my gateway, right? So I can just focus on the main stuff that mall does. So that's what we're talking about, two packages. And ellmer does a great job at this kind of integration as Libby's going to show.
So I just want to mention that. No, yeah. And I want to hop in and answer David Onder's question here, which was what kind of specs should I realistically have to make good use of these models? And he said he had 64 gigs of RAM, more than enough. 64 gigs of RAM is more than enough. You would be fine with 16 gigs of RAM, I think for 3.2. It's not as big as you think. But also when you go to the Olama download and you look at the different models, each of the models actually has like two or three different size versions of that model to install. So go take a look at them and see what you would need for each of those.
Setting up chat objects and using mall
All right. So I have created my chat objects for both Anthropic and Olama. Yours might look something more like this, like chat, chat open AI, and then model equals. Don't worry if you do not have something that looks like this. This is very specific to my setup on my machine for work. Okay. So I need to tell mall I have this ellmer chat object. I want to use it. I want to use Olama. So I'm going to run use Olama. Actually, I feel like I should throw a .cache in there. So I'm going to do that. This is me trying to stop a cache problem from building up because I'm trying to break the ellmer package here. So I will do this and tell mall I want to use ellmer on the back end. Yes. And then my LLM session, my model is using Llama 3.2. We're all good to go. Now what I'm going to do is I'm going to apply the LLM sentiment function to my content variable.
So let's go look at my tweets sample. This is the data set that I'm going to use right here. If we take a look at it here in the console, I only have two columns, author and content. Author is a Twitter screen name and Twitter handle and content is just the contents of the tweet as a string.
So I'm passing the content variable right here to the LLM sentiment and then I'm giving it some extra optional things. So let's go over to mall really quickly and we'll go to the sentiment function here. The R version because that's what I'm using. And we're going to go to its R reference page so we can see what we have as options here. We've already given it the dot data because I'm piping it in. Remember when you were piping something in, that pipe, it takes everything on the left hand side of the pipe and feeds it to the next function as the very first argument. That means that this is satisfied. Then I'm giving it my column. That's that content column. And then you can give it options. This is like, hey, these are the options that I want to accept as output from the LLM. Mall is very smart and if it gets output that doesn't fit that, it will coerce it to NA for you. So take a look at that. And then pred name, most important one to me. I don't really love the dot sentiment dot pred dot whatever. I like naming it something specific, especially because I like to compare different models. So I'm going to change this. And then you also have the option of an additional prompt. I wanted to ask you, Edgar, is this like a system prompt or is this an additional individual prompt?
This gets attached to the main prompt that's being sent. So if you, for example, something like this, you can say if the person says these specific words, then consider it neutral or whatever. So you can add some extra feedback to make it more fine-tuned and better.
But I am not going to use that. What I am going to do is just say, hey, here are your options, positive, negative, and neutral. And name my output column sentiment llama. So I can run this. And if everyone has lit their candles correctly, my code will run. A little progress bar at the bottom here. Love that. Aha, there we go. It was created. So I can go look at my tweet sample sentiments, which is right here. Click this button to get a data view.
And here we are. I now have an extra column over here of sentiment. This is only 25 rows, y'all. If you have a enormous data set, please be prepared to wait. And please be prepared for your laptop to go for like 10 minutes while it figures out what it's doing. Because it really is your machine just like cooking, okay?
So I had a question in Discord. Are you limited to those three options, positive, negative, neutral? I could give it whatever I wanted. So you could do something, yes, like very positive, positive, neutral, negative. I would suggest giving it instructions about what each of those things means, right? Yeah, that would be important. Like if you're deviating from that kind of basic three structure or even two structure, I would let it know what you mean by very positive. We will see in just a minute. I'm going to compare the outputs of Olama and Anthropic. And you can be the judge on how well each of them did and whether or not they would have been coded the way you would as a human.
Libby. Yeah. I think Javier is also asking if it can return the prediction probabilities, which doesn't because it's not part of the prompt. And ellmer, like if you go into the articles themselves, there's a way that you can actually build a prompt like that where you say, you know, give me a percentage you think that you can kind of make it do that. But it's not, I don't think you'll be able to get probabilities properly as if it was like an actual LLP model.
Also, while I have the floor, I just want to mention real quickly, because of the time, how long it takes and know that that's what we have the cache option. So like if you're running the same, you know, QMD or same script, and you just refine in it, and you're going to get the exact same result. If you were to run this, let's say LLM sentiment, then it works much faster the second time because it's automatically caching results. Into a temporary folder that was faster for you, you don't have to rerun everything. So that's why that cache exists. Just want to mention that. Yeah, that's if you want to save some time. That's why I'm giving it the cache null. I really don't want it to build up in there.
We had some questions in the chat, not all of them I can answer. So Abdullah had asked a rule of thumb set of specs for a machine. So local LLMs are practically feasible. And I mentioned that if you have at least 16 gigs of RAM, I think that you are you will do okay with something like 3.2 from Olama. I am not a technical spec expert. Don't listen to me. I would have some people add in the chat what their specs are on their machine and how their machine does when they run things. I have found that my machine does okay. But your mileage may vary based on all kinds of factors.
Brent Pfeiffer had asked, would that be in the additional prompt area where you're defining things? Yes. Where you're giving it information on like what extra negative is or extra positive is or in between is. And then Simon had asked, let's see. Ah, Simone, you had asked, you could use the vitals package to compare models, right? The vitals package is awesome. That's Simon Couch's package, right, Isabella? I have never used vitals, but you could. And then can we use LLM Studio instead of Olama? David, I do not know. I'd have never used LLM Studio. If anybody has used a third party LLM management system, let us know in the chat. That's not something that I have played with yet.
Comparing Olama and Anthropic sentiment outputs
All right. So let's go ahead and switch our models over to Anthropic. That means I'm using the mall LLM underscore use to switch from that chat Olama to chat Anthropic. And again, using the cache option there to say, hey, don't fill up my cache. And then what we'll do is we'll run that exact same code with LLM sentiment, same code. The only thing that's different is I'm changing the column name that I'm adding to sentiment Anthropic. So I'm adding it back to that same new data frame that I had created up here. So I can run this. Let's see how quickly this runs with Anthropic. In all of my testing, it's taken the same amount of time, almost exactly, for Anthropic to run versus Olama, by the way.
So now I can, it's good to read all of these many things I have open. We will go open tweets sample sentiment one more time. And this will allow us to look at a comparison here between these two. The one on the left is Olama. The one on the right is Anthropic. And we can see that they don't agree for all of these, right? So this is Olama. And Olama is coding, for example, this is negative when all it says is Mondays and fur babies. Mondays could be negative for sure. Like Olama, you might be right. But Anthropic coded that as positive. We have another disagreement on the second one that just says tomorrow Toronto and a URL. And that's it. I've noticed that Olama frequently codes something that has a URL in it as negative. And I don't know why. But that seems to happen a lot. Whereas I think that Anthropic is more correct here that this is a neutral tweet. None of us know the context of this. We can't see what that URL goes to. And if there was a photo, we can't see it.
So let's head back over to our code. And let's look at our agreement here. What I'm doing is I am looking at this sample of sentiments, which I just showed you here. This is so that I can render the document. And you can look at it as a PDF if you want to. And then I calculate the agreement. So 0.68, 68% of the time, they agreed. Otherwise, they did not match. This is something where I really recommend you go through and review. We all know that LLMs aren't perfect. Neither is NLP, by the way. If you code something through NLP, you might look through it and disagree as well, even though you wrote the underlying structures of your NLP model.
We all know that LLMs aren't perfect. Neither is NLP, by the way. If you code something through NLP, you might look through it and disagree as well, even though you wrote the underlying structures of your NLP model.
Validating model outputs
Let's head over to summarization because we are about halfway through. I want to make sure that we don't have any major questions. Let's see.
I see a question from Abdullah. How do you iterate and validate the model? Let's say you don't agree with how the model classified things. How do you measure accuracy? Do you take a random sample of results and let a human evaluate it? This is a process question that will depend on how much you want to verify. I am playing with tweets. I'm just going to visually verify, cherry pick, and go based off of vibes, right? But if I was working in a research setting where I really needed things to be right, I think that I would probably run things through an LLM. And then I would set up a verification system to have several different human beings run through and validate whether or not this was correct. So they will not have to read a tweet and decide whether or not it is positive, negative, or neutral. They will just have to agree or disagree with the model, what the model had coded it. And if it was wrong, they might recode it. And then I would have human beings reviewing those results. So depending on the way that you do things and how big of a project this is and what the stakes are for the project, that'll be a choice that's up to you. Maybe some people can share in the chat how they have validated things.
Simon Couch's blog is a great place to go for all things LLM verification, by the way. I would go read through that. He's going to be much better at that than I am. OK, Isabella, can you stop me if there are any other really big questions? But if they're not, OK, I'm going to show you some summarizing really quickly. We're going to go through this one a
little bit faster. I'm switching over to LLAMA 3.2 again. And I'm going to use Jimmy Fallon tweets this time. And what I'm going to do is create an extra column in this new tweets underscore Jimmy Fallon underscore summary data set called Summary LLAMA. And the extra thing I've given it this time is a max words 10. Now, LLM's never listened to us. It's frequently not going to stop at 10 or give me less than 10. But a lot of times this can help stop it be too wordy.
So let's head over and just redo this with Anthropic so that we can compare these.
All right, here we go. Anthropic is running again pretty quickly for a summarization task. But also, this is only 25 rows. Let's go look at it. Go over to my data here. Check out my tweets Jimmy Fallon underscore summary and open it up here. All right, here's my here's my content.
My friend slept in our walk-in pantry. When he laid down, his feet would stick out the door. Hashtag my first apartment. Okay, LLAMA's on the left.
Crazy and quite concerning living situation for a first apartment. Not wrong, technically. Let's see what the Anthropic summary is. Friend slept in tiny pantry, feet stuck outdoor. That is much more accurate to me, that Anthropic summary, right?
Always great to see Mets Bucket Hat Guy, a.k.a. Mike DiCenzo. DiCenzo. The Llama Summary, Mike DiCenzo is a New York Mets fan Twitter personality. Not wrong! Met's bucket hat guy spotted at event probably more correct.
So again, we have this like Olama is not technically wrong, but is maybe not super helpful and then anthropic is slightly more helpful. You're gonna have to go through and review whether or not this is good enough for you, right if you're doing a summary task with Olama locally. Is this good enough? If not, you might want to attempt to lean on a more commercial LLM like anthropics models.
I'm pausing for the chat. Okay. I don't see any major golf flags, which denote our questions, so I'm gonna head back over and we just did this. We just took a look at what they were.
Custom prompts and classification
Now I'm going to go through really quickly because we have 20 minutes left and I want to stop talking and hand this over to show Python code. What if you don't like those standard functions, what if you want to do your own thing? Let me show you how to make a multi-stepped prompt using the anthropic model specifically.
What I'm gonna do is look at Katy Perry's tweets and have it classified for me whether or not they are safe or dangerous and I am going to define safe or dangerous. Really simply here and I'm also going to tell it to ignore URLs because I've noticed that LLMs just get like confused by out by URLs sometimes. So I'm going to say ignore URLs and usernames.
I'm also going to tell it safe text contains no slurs or curse words and I'm going to tell it assess whether the text is safe or dangerous and then return one word either safe or dangerous. I am saving this in a prompt object. It's just a string it's not going anywhere yet. It's just a string object saved as prompt.
I'm going to switch back over to Anthropic, which I think I did before. I think anthropic is the last model I used. Yeah. Okay, so I'm gonna leave that. And then I'm gonna go through my sampling of Katy Perry tweets. For my prompt option in the LLM underscore custom mall function I'm gonna give it my prompt and then I'm also going to give it what my valid responses are only safe and dangerous. I want it to course and everything else to na and then I'm going to name my new column that it's creating safety underscore anthropic. So let's create this see how long it takes to run through with that custom prompt.
About the same time as it's taken everything else at a larger scale. The differences in time are going to be much more apparent. And let's go over to tweet safety look at it with our data viewer and positron.
And oh, wow, we do not have the correct author here. Did I use the right data set. Oh, wait, I need the new one that I created. I need tweets safety. Did I do that one? Oh, what have I done here? Y'all help me.
tweet sample oh, we're looking at safety of just like regular tweets first, I forgot I did this they're all safe. The random sampling that we took which is like all kinds of people Rihanna Kim Kardashian Katy Perry Ariana Grande, they're all pretty safe if we're looking through these. I don't see anything that leaps out to me as something that I would have wanted it to flag as dangerous.
Does anybody else see anything I do not see anything no slurs no curse words.
Okay, was I smart, did I do Katy Perry's tweets? Yes. Totally random assortment not random. I went and found bad ones. Here's what we're gonna do. We are going to apply that same exact prompt to Katy Perry tweets. We're gonna create a data frame called tweets Katy Perry safety. It's working a little bit harder on Katy Perry here, we're at 89%.
Did everybody light their candle.
Okay, let me check the chat for questions y'all. There is one question about why use paste if it is all text. Remember what line that was on? Ah, that was me creating my prompt up here. I used paste because this is exactly taken from the docs for mall and this is just what what was in the docs in mall and that's all I knew. So I'm just going to copy and paste.
Oh, that's what the question I couldn't figure out where the question was from yeah. You can just put this all together. What paste is going to do is put this together with a space between each one you can do paste zero and have no spaces like you're in control. But having it like this is nice because it allows you to look at it line by line and understand what exactly you are telling your llm to do. I do like the visual separation of it.
Um, yes, dan had asked are these recent x no, they're tweets. They're literal tweets. This is from the before times y'all. Um, the data set is on kaggle. It's the tweets data set and isabella had shared a link.
Okay. No more questions. Isabella lit a candle for us and it finished coincidence you decide. Okay, let's go look at our Katie perry safety. Do we dare look at this live on the internet? We're gonna do it.
Okay, katie perry's tweets they have been classified as safe or dangerous this first one is safe. A visual explanation to what people in florida are in surrounding areas experiencing send them prayers looks pretty safe.
Row three here has it has decided this is unsafe and it does have a curse word in it. I think that that is great. Good job anthropic. You did a good job.
Okay, so before I move on I want to say that prompt took iteration. It did not work the first like eight times. I did it. It didn't do what I wanted it to do. I had to iterate and iterate and iterate until my prompt was going to do what I wanted it to do reliably.
Okay, so before I move on I want to say that prompt took iteration. It did not work the first like eight times. I did it. It didn't do what I wanted it to do. I had to iterate and iterate and iterate until my prompt was going to do what I wanted it to do reliably.
Also, um llms are not deterministic. I could run it one time and run it again, and it could you know, give me a different answer. So use llms for things like this for anything at your own risk here.
Um, all right, we ran that. I also just for funsies because everybody likes numbers looked at my random sampling of 18 katie perry tweets to see what proportions were safe and unsafe. You know about 83 percent safe not bad.
Token usage and handoff to Python
I wanted to also before I hand this over for the last 15 minutes to edgar here talk about an ellmer not mall an ellmer function, which is pretty nice. It's the token usage function. I think this is pretty new right edgar. It tells you how many tokens you've used. So hey, here are the the models that you've used here are the number of input and output tokens.
Notice that I don't have anything that's cached because I told it not to cash anything. I think that's how that works. And then the price is in a I have seen people where it has access to a price for an actual commercial model. This is me using it through bedrock so it's not the same. But I have seen people use this when it's hooked up to a commercial model where the price actually does show.
Edgar, do you have anything to add about that function? No, not really. I just noticed I haven't really used it that often.
So, um, cory had asked playlist for the data science labs. Um, isabella linked it but cory and anybody else listening if you go to pos.it slash ds lab. You will also see the playlist linked there. It's like the registration page for our our event, but it also has a bunch of other info.
All right. Yes. Yes. Yes. Did I miss any questions? I don't think I did. I'm gonna hand it over. Let's stop sharing here. To edgar and edgar is going to show you um what the python version of this might look like.
And hopefully if we've all written we've all lit our candles correctly, um all right, yeah, you're his code. Yeah, there we go. There you go. All right, so is this uh, how long do I have? All right.
Um, I uh, i'm assuming that most of us here are our users. Um, so i'm going to do some explanations here like additional explanation of how it actually translates into into python. Uh, actually how mall workflow translates into python, uh, so that's hopefully can well you can also see as an r user how how it differs in this thing with uh python.
So mall unlike r where you can have like a standalone uh package or something that um can modify data frames, uh in python, but specifically for the pi data community uh, we have something such as pandas and that has some pandas data frame for example. Um, but uh, there's a new one called polars and let me see what's it doing this? There you go. Polars.
You know, just switch here. Uh polars is um it's great. It's it's very fast. It's been it was written in rust. And this was uh, a lot of uh folks are starting to use now, in fact, it's like the recommended uh way of using data frames going forward. Uh, so that's um what mall does on the python side.
Uh, i'm gonna stop sharing here. Um, let's see, sorry, uh okay, so um you can see yourselves, okay. Uh, we see okay, there we go. Okay, were you able to see yourselves or it just goes black goes black. Okay, good. All right. So i'll just keep it like that for now.
So what it what mall is is it's an extension of polars and clarify here what extension what I mean by extension here in a second.
All right, we started with the errors here. There you go. Now uh, I have this uh critics data frame loaded is uh, these are criticisms, I guess uh or reviews about the Nintendo game animal crossing new horizons. So can be pretty favorable um using the Tidy Tuesday uh data sets. We'll have a link here so you can see essentially, uh the name and also the actual uh text of the review.
Uh, the next thing i'm going to do i'm going to import mall the field that actually notice that but go to critics and type dot which is kind of like dollar sign in our if I start typing llm you see that's not available. And leave me short earlier where you can tap you can see where llm dot sentiment that that's not available. Becomes available whenever you actually load the mall package. Now you should be able to see it.
Which was something really interesting to me because i'm kind of new to python. So that's basically what extension means is basically that it becomes part of that um object that you have. Which is your data frame is an object that now is extended to be able to use uh the functionality so now that it it's there uh we have to assign it per data frame uh, which back in that we're going to use as a provider and i'm going to use olama here.
So once you start mall, it's actually going to install this package too And then I can just use it And very similar to how it works where instead of piping into it just basically call the extension and run it
Now it's gonna it's gonna run Yeah, so his little call there on line 34 is the data frame name dot llm dot sentiment so we're Working on that data frame class And then we're passing the text which is a variable name as a string
correct For anybody who's not used to looking at python That's something that kind of Took me a while to get used to where I have to always quote the names of the of the actual columns i'm using So it's going to be quoted Uh, so it's running right now Hopefully it won't take that much longer
In the meantime, we have a question from dan asking In the documentation says in python mall is the library extension to polars Does that mean that it does not work with pandas? Correct you won't be able if it's a if it's an actual pandas data frame And you don't have polars loaded. It won't recognize it. You won't be able to use it. You have to convert it
Yeah, it's taking a while And I don't have the time Yeah, please eight minutes left Okay Wait candles. Oh my gosh candles. Yeah, we need them One more it worked last time It worked last time This is all i'm saying. I know it's running because if not, it would uh air out and I can see The red square. Yeah, so everything is it's working
Uh, but i'm sure it's having all these screens open and zoom and all this stuff Just uh making it Chug along. Come on, you can do this Megan gave us a very nice little explainer in there about python objects and extensions and methods and stuff Yeah
so Well, this is doing this i'm gonna Hubert see you later. Some folks are having to drop for other meetings. That's okay
Exploring the mall documentation and cache
I want to show here. Um as laby mentioned there's um Also, you can select arm pipe or python here and it can walk you through how you you can set that up in your uh in your machine Uh, all the same examples are available as well as the reference Once you select python and any of the pages in the site, it'll switch everything to python. So the Documentation will also be available for for it
um The package itself on both sides on our and python side come with a small data set that has three reviews and uh You can use it as you know to test it for the first time and uh It's all available here, too
perfect, so Let me know do you finally That our python has run in our last Yeah, I got I got the I put it on my other screen while running. Yeah, so you can see It ran and if I were to run this again You notice that it ran almost immediately because of the of the cache So you can see the big advantage of having the cache is especially if you're trying to re-render things and rerun them It's it's not going to take as long the second time
So you can see the big advantage of having the cache is especially if you're trying to re-render things and rerun them It's it's not going to take as long the second time
So i'm just going to reassign this variable here. So now we'll have the new column Which is the sentiment column? We'll have it available And then i'm going to run Uh to see if there's anything that is not possible because everybody liked it This is the only one that kind of Didn't like it uh, so we can see there's some variance there as far as Uh, not everything was classified as positive
Extracting language from user reviews
um for this one i'm going to use the The first one were critic reviews. This is the user reviews. I'm going to read this in And instead of 100 i'm just going to do the top 10 for right now And what i'm going to do here is i'm going to extract the language that the review was written in And that should take shorter hopefully Hopefully with only 10. So we're using llm dot extract or llm underscore extract from r We are passing it the text variable and then the prompt that we're giving it basically is I want you to extract the language Be it english spanish, etc. So that little part at the end that second argument is what the llm is getting as its prompt
And hopefully it does okay because we do have some reviews in all different languages Yeah, good question. Edgar. Where is this data from? This is uh from uh, teddy tuesday. It's 2020. Uh, may 5th awesome Thank you
Um, I don't think we're going to have enough time to go through all this because it's taking a while um, one thing I wanted to show is uh the dust translations There's a few here that are not uh in english that could kind of showcase Uh, the translate actually does refer to human language so you can translate from one language to the other um, one thing that I found with llms that is so cool is that you don't need need to specify the Uh origin language just the target language so it'll adapt to whatever language you say that you have
uh, you know Italian spanish russian all in the original text that you want them all translated to english You don't need to specify each one as a you know, the origin just to say I want english which is very different from other um translations uh, you know Programs or algorithms that are out there. Uh, the elements just pick it up and it does it automatically And this is true both sides on the art and python side
um, one thing that I found with llms that is so cool is that you don't need need to specify the Uh origin language just the target language so it'll adapt to whatever language you say that you have uh, you know Italian spanish russian all in the original text that you want them all translated to english
Using an external LLM provider with chatlas
Uh before we go. I just want to show you uh how Uh mall works with an external uh llm or external provider Uh, so instead of uh ellmer we're using chatlas, which is the essentially the same package but for python Um, these are this is not my package, uh or ellmer for that matter ellmer's is uh, at least package
uh, so I'm basically doing the same thing. Uh as Let me do it on the other side. We're uh setting up a chat object And what i'm doing here is under user reviews. I'm going to use that chat object So now it's set up For me to use and I can use it directly here Now it is calling um uh aws is going to uh anthropic and running it just like it ran with uh, uh in our The only thing that we don't have is the nice progress bar But uh, it is working. So oh wow on this one said it's all negatives This is from user reviews, yeah, but it worked it looks like but it works. Yeah, some of them are right
Closing remarks
All right, everybody We have a minute left um, I wanted to re-put the Repo in the chat that um, we use that contains both of these files. This is just my personal repo if it's a mess Don't judge me. This was really really fun I'm, so glad that you hung out with us at the data science lab
I wanted to let you know that we have sarah altman joining us next week And we just happen to have another llm one. The lab is not always about llms I promise but we are going to be doing data analysis With the assistance of ai which might mean data bot might mean clod code. Like what does that look like in? February of 2026 for sarah. I hope that you'll come and join us and on thursday at the data science lab We have alexander shock from sitel, but he's also the effective statistician podcaster Come with your data science career questions, especially if you are a stats flavored data science Person, it's going to be transformative and wonderful
I hope that you have a fantastic rest of your week Please hang out on the discord server with us if you have more questions. We are going to try to answer them We love you, and we'll see you next time. Bye everybody

