image: thumbnail.jpg
Jeroen Janssens - Package Your Python Code as a CLI | PyData London 25
Learn how to transform your Python code into a command-line tool. Jeroen Janssens, author of Data Science at the Command Line, guides you through the process of turning your scripts into reusable, executable tools, integrating them into your data workflows and harnessing the power of the Unix command line.
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to this tutorial where we're going to examine how you can package or transform or convert your Python code into a command line tool. My name is Jeroen. I already got a great introduction so we can keep this brief. A lot of the material today is based on a book I wrote a couple of years ago, Data Science at the Command Line, which you can read for free. If you go to my website, you can easily find that book. A couple of months ago, another book that I wrote came out together with Thijs. What's good to mention is that we're giving away 25 signed hard copies on Sunday. If you want to participate in the raffle, you can go to this URL or scan the QR code. If you then fill in your first name and your email address, then you get a chance to win a copy. The winners, the lucky winners, will be announced at tomorrow end of day. That's polarsguy.com.pydata.london.
For the next 90 minutes, we're not going to be focusing on Polars. We're going to be focusing on that other technology which has been around for a bit longer, the command line. I'm not going to explain why you should use the command line. The fact that you're here pretty much shows that you are already convinced that the command line is a wonderful way of working, very complementary to using an IDE or a programming language. One of the beautiful things about the command line is that you can very easily extend it. You can create your own tools. There are already a lot of tools available. Even if you just install a Linux distribution or if you have Mac OS, there are, I'd say, about 100 tools already at your disposal. Then there are thousands more tools available to you. New tools are being written every day. The good thing is that it's actually not that hard to write your own. That's what we're going to be looking at today.
I really believe that if you're able to create your own tools, that they'll eventually, or actually immediately, make you more efficient and productive and effective. That's, of course, a good thing. Because the command line has been around for so long, I think I have another slide about this too. Because it's been around for over 50 years, we can safely say that it's going to be around for the rest of your career. I hope it will be a long and prosperous career. And there will be many, many tools, many programming languages will come and go. Plenty of JavaScript frameworks. But there's this one technology which is going to be sticking around for a very long time, and that is the command line.
But there's this one technology which is going to be sticking around for a very long time, and that is the command line.
Overview of today's tutorial
So this is roughly what we're going to be covering today. First of all, I want to make clear what is it that we're actually talking about when we say CLI. Then the interface. How do you design your next tool? And there are two components to that. On the one hand, arguments and options and switches. And on the other hand, how do you deal with streams, streams of data? Then we're going to go into the fundamentals. How can you transform an existing piece of Python code into something that is usable from the command line? In six easy steps. Then I'll show you how you can upgrade from that, you know, the fundamental way of doing things using either R parse, Python module, or Typer, a relatively new Python package that you can use to easily build modern command line tools. We'll have a look at UV and how you can use it to create self-contained tools. And along the way, I'll share some best practices.
This is a tutorial, and I really want to encourage you to ask questions along the way, right? So this is your opportunity to extract as much information as you want from me. We can, you know, I've prepared plenty of slides, I have some exercises, I really want this to be interactive. So any question is encouraged and welcome.
What we mean by the Unix command line
All right then, what are we talking about here? We're talking about this. We're talking about the Unix command line. So if you're running either Mac OS or a Linux distribution, the most popular one being Ubuntu, then you're good. If you're currently running on just Windows, that's not enough, right? Windows does offer, you know, two types of command lines, but they're, you know, a different breed of command line interfaces. So if you want to follow along today and you're running Windows, you have two options. You can either install Windows subsystem for Linux, WSL, or you can use Docker and use a Docker image that has a Linux distribution on it. And at the end of this presentation, there are links to this, or actually also in the description of this tutorial on the PyData website in the schedule, there are links to, you know, how you can follow, links to instructions and how you can either install WSL or use a Docker image.
Is Git Bash going to do the same thing? That's a good point and it might actually be good, good enough. If it works, it works. Great. Yes. So thank you very much for mentioning that. If you have Git Bash installed, then that works as well. Thank you.
We're not going to be talking about textual user interfaces or TUIs. They're great, right? Here we have LazyGit, for example, but they are a very different approach. They're more like applications that, you know, are running in your terminal, but they're a very different approach than working with command line tools. If you are interested in this and you are interested in creating a TUI using Python, then I can recommend you check out the textual package.
Looking at a simple Python script
All right. Let's have a look. So this, right. We're at the command line. I guess we want this to be a little bit larger. Yeah. Is the text large enough for everybody to read?
So I have indeed spent some time on customizing my environment, my prompt. And please don't be intimidated or anything. This is totally not needed for today. Yeah. If you are interested in this, we can talk about this after today's tutorial. Yeah. I'll be around for the rest of the day and the entire conference actually. I'm always happy to talk about this. And I am working inside Tmux, right, a terminal multiplexer so that I can easily switch between different windows and panes.
Okay. Yes. I was going to show you this script, right? What was it? Yes. I'm going to be running a script called greet simple.
There we are. This is the output of that script, right? What does the script look like? Let's have a look here. So this is not a command line tool. This is just a very small Python script. Yeah. It opens up names.txt. And then for every name that is in this file, it will, you know, print it out along with a greeting. Yes. This is what names.txt looks like. Yeah. Just 10 random names.
As simple as this is, right, it's not so much that it's simple. What it represents is it's a script that may very much be like something you have created in the past. Yeah. So it is something that we can turn into a command line tool. Yeah. Using a couple of steps. But there are certain aspects here that need to change. For example, there are some hard-coded values in here, such as the file name or such as the greeting. And it doesn't offer any, so it doesn't offer any flexibility. It also doesn't provide any help. You know, things that we come to expect from a command line tool. Yeah. So what we're going to do today is turn this into something that we can actually call a command line tool. It can be run from the command line already, but only because I invoked Python, which then interpreted this script.
What is a command line tool?
So when I say command line tool, yeah? It's an umbrella term. We have a variety of different tools and how they're implemented. Yeah? We can have simple aliases or shell functions defined in our configuration file, or we can even have scripts that are compiled and so binary executables. We're going to be looking at interpreted scripts, right? Because of course today our language of choice is Python. Yeah? And I believe that this really hits a sweet spot when it comes to creating your own tools. They give you all the flexibility you need from a programming language and still all the power, maybe not the speed, but still all the power from a binary executable. And a lot more flexibility than a small shell function or alias provides.
Yeah? But the good thing is, and you might already be aware of this, is that the command line itself doesn't really care in which language a script is programmed in. Yeah? As long as you adhere to certain standards. Yeah? It doesn't matter whether it's in Bash or Python or Rust. We do see more and more command line tools being implemented in Rust. Yeah? That's a very welcome thing. But this also means is that you can easily combine tools written in different programming languages. Yeah? So if you want to take a polyglot approach for whatever reason, maybe you're working with a team that works in another language. Yeah? Then this is a way to do that using the command line. So. And then there's this other part of the so-called Unix philosophy. Yeah? Is that most tools, most, not every tool, they do one thing. But they do it really well.
Yeah? Because you can string these tools together that gives you a lot of flexibility. You can solve many, many different problems that way. So keep that in mind as you are designing your next command line tool. So you shouldn't try to implement everything in your own tool. It's good to be aware of the existing tools that are already out there. Yeah? So these can be features that the shell provides, such as piping or reading from a file, writing to a file, or globbing. Yeah? So finding multiple files using some pattern. Or command line tools such as cat curl for downloading things. Yeah? If your command line tool downloads something in order to process it, you may want to consider, like, hey, how about I just rip that out of my tool and let curl them do the work so that it can pipe its output to my input. Yeah? So that requires a little bit of a different way of thinking.
Yeah? So in other words, stay in your lane. Try to focus on what your tool needs to do well.
Yeah? So in other words, stay in your lane. Try to focus on what your tool needs to do well.
Thinking about your next command line tool
So here we are with our very first exercise. Don't worry, it's not a coding exercise. This is a pen and paper and a thinking exercise. Although with coding, I do think that, you know, we still do a little bit of thinking, right? So pen and paper. I would like you to think about what your next command line tool could be. So try to remember something that you have created in the past, try to remember something that you have created in the past, whether that was a helper script or a one-off notebook. Does it contain something that can be generalized, right? If you strip out all of the hard-coded values. So what was a particular task that you may be, that you have to do every so often or that you may be needing to do in the future? And then let's think about how you can turn it into a command line tool. And I know we haven't talked about the interface just yet. For now, I just want you to think about, is there already something out there that you have created that you can use? And if you cannot really come up, can come up with something, that's all right. Maybe you can think of a tool that you may want to be designing.
Is there something in your work or otherwise that requires some tasks that you cannot really comfortably do using the existing command line tools already out there? So that might be a gap, an opportunity for you to write your own command line tool. So I'm going to give you a couple of minutes for that. So pen and paper or make notes on your laptop to think about what your next tool could be.
Pen and paper or make notes on your laptop to think about what your next tool could be.
Try to focus on your exercise, I'll try to focus on this setup that I've never encountered before, these issues. I guess I said all of these things, right?
not for everybody but more to your personal setup at the moment. Now would also be a good time, either Thijs or myself might be able to help you, whether that's regarding your setup or any other question that you have.
Yeah, you want me to come over or do you think it's a, would you like me to come over or Thijs?
I have this repost from my work. It's a mix of a Quarto doc that then builds the GitHub Markdown. This is the Python script and I was thinking to make this into a standalone tool and it would take these parameters like a vehicle.
Wonderful. Yes. So ideally my co-workers could use it rather than me having to run it all the time. That would be my basic idea. That sounds wonderful. Yeah, excellent. Yeah, that is great, great.
Designing the command line interface
All right, let's continue. Don't worry if you're still thinking or writing. Okay, so you have a tool in mind. We don't know yet what it's going to look like but that will come in a moment. Let's talk about how we can design the interface of our next command line tool. Like I said, the interface is roughly based on two aspects. On one hand we can give our tool arguments, also known as options and switches and flags, and on the other hand we can let our tool deal with streaming input or input and output and error messages. So let's first focus on those arguments. Now, there are a lot of terms here when it comes to those arguments and various sources use different lingo. But this is what I believe to be the case. So here we have one example.
This git commit command. Now, git is what is the command. Commit, in this case, is the subcommand. Dash v, which stands for for Bose, is an option. Or, no, it's a switch. You see, it doesn't really matter what we call these things. Just know that, you know, sometimes they are called options or switches or flags. I believe because this particular one, so dash v, doesn't take any arguments, any parameters, it can be known as a flag or a switch. And then dash m would be an option. And in this case, yeah, dash m, well, they're both optional, yeah, but you can have optional arguments or required arguments. And so everything besides git itself, you know, git is the command, and all of them are arguments, yeah. So that is the top-level name for all of these things. And to demonstrate that, I've created a very small tool that we can run from the canal line. Yes, you see this. Fantastic.
Yeah, so we have here print args.py. So, well, we can, I guess we can mimic what was there. And what it prints is a list, a list of strings. Text is the universal interface on a command line, so all the arguments that you give are just strings. If you need to treat them differently, you may need to do some extra work, especially when you're using, yeah, the built-in functionality that we're going to start with in a moment, yeah. Other modules and packages may be able to do some work for you. The very first value in this list, known as argv, is the name of the tool, the command that you used. So if there was an alias, yeah, that was pointing to this tool, we would know. That would be, that name would then appear as the first value in this list. So we have commit, we have dash v, dash m, and then fixed hypo is one value because I used quotes, yeah. If I didn't use those, they were, they would be, you know, separate, separate values.
Yeah. What's interesting, by the way, is that I can also show you, let's say, so what I did here, the very last argument is, I used something here called command substitution. This is a feature that your shell, the dash or the z-shell or fish, yeah, takes care of. So this is not something that your tool has to do, yeah. It's great. Same as with, with globbing. If I don't use globbing, I don't need to Yeah, it's great. Same as with, with globbing. If I were to do star dot or asterisk dot pi, we would get all the file names as separate arguments.
That's neat, huh? That's neat. It's good to know something that you, it's already functionality that you don't have to worry about.
So, oh, wait, I guess I can show you the code. Oh, it's already up here. Fantastic. Yeah. So a very short piece of code. It just preprints the argv value. So arg values. What are the arguments passed to this script?
And I'll explain later what this very first line is. It's crucial. It's crucial. It's crucial. It's crucial for creating a command line tool yourself, for any command line tool, really.
So we use sys.argv in this case. This is as low level as you can get. Just a list of command line arguments. If you want to do any fancy things, then if you want to support both short and long options, if you want to do any verification, then those are things that you have to do yourself. So that might be a lot of work. But if you're creating a tool that creates zero, one, or two, or that needs zero, one, or two arguments, then I would say that's fine. If this tool is for your purposes only, and I would say that's 99% of the tools that I expect you to create, then it's fine to use argv. So the first element is the script name, and the remaining elements are arguments. And another reason why it's good to be aware of this is that the other solutions that we're going to be looking at, the arc parse module and typer, or maybe you want to use click instead, or something else that I've never heard of, all of these Python solutions, they still use argv under the hood.
Command line option conventions
So when it comes to options that you can use for your next command line tool, there are certain conventions out here. And a lot of tools, not all of them, there are always exceptions. I mean, that happens when something has been around for that long. There are always exceptions. But it is generally understood that these short and long options, they're common. They're common conventions here. So when it comes to providing help, you would provide both the short option dash h or the long option dash dash help.
I'm not going to be listing them all. So they're more for you to consult as a reference as you are when we go to the next exercise.
So dash f if you want to specify a file or if you, you know, just take a moment to look at these.
And again, there are always exceptions. I would say that, you know, with the next exercise, don't go overboard. It's not necessary to have all of these implemented. And I would say the fewer, the better.
So when it comes to output, outputting something in JSON is very popular these days.
Maybe you want to generate some configuration file automatically. I don't know. That you can first inspect in some format or maybe, you know, output it as JSON when you want to save that. I don't know. Just making this up. And now it's your turn to make things up.
So I want you to think about the arguments that your next command line tool will get. So try to think about, okay, what is their purpose? Do they change a setting that is crucial? So are they required or are they actually optional? Do you want to control the amount of logging that your tool is going to produce? So optional or required. Yeah, just think about it. And try to come up with a couple of invocations.
Does your tool need subcommands? It's fancy, right? I mean, something like Git has a lot of subcommands. But is that really something that your tool is going to need in order to get the job done? Yeah, something to think about. Again, I'm going to give you a couple of minutes to just brainstorm about this.
Of course. Shall I come over? Or is it a general question? Good.
All right. Yeah. Okay. So what are my thoughts when it comes to short options and long options?
So Go violates, I would say, the convention that for long options you use two dashes. And I think that it's not a common thing. It's an exception, I would say. So I'm not a big fan of that, personally. No. So what happens then? So these short options, many tools allow you to merge short options together. So you would use one dash, and then you can have a couple of letters. And they would each be short options that you could provide separately. So for example, tar-xzvf. I read that xkcd comic just this morning to remember the arguments to tar. You can paste them together. But how does Go then handle that when also long options only need a single dash? Yeah. So yeah. Good point. But so this is, again, like I said, there are always exceptions to whatever we're discussing today.
Command line tools and standard streams
Shall we move on? Yes. In a way, command line tools are kind of like functions. Yeah. Why do we use functions in our code? Well, to abstract away some complexity. Yeah. And functions can take zero, one, or more arguments so that you give them some flexibility, that they do one thing and do it well, hopefully. Right? But there is at least one thing that sets command line tools apart from your Python functions. And that is that each process in a Unix-like system has three standard streams available.
And again, it's all text. Yeah. That's the universal interface. So if you want to output numbers or if you want to ingest numbers from some other tool or from a file, you need to do some work there. But that's a necessary thing in order to make this a general interface that always works. So there are three standard streams. Standard because they're always available. Standard input, standard output, and standard error, abbreviated often as stdin, stdout, and stdir. Not every tool uses these. Your tool may not need it, right? But that's something that we're going to think about in a moment. For example, pwd, print working directory, doesn't need your input. It just prints out the current working directory. And if you haven't done anything special, it will be printed to the screen.
Rev can read from or does read from standard input. It reverses. It inputs line by line. And if you haven't done anything special, then it will read from the things that you type. It's attached to your keyboard input. So let me demonstrate this.
So I'm just typing rev here. And you see this cursor? It's not doing it. There's no output. It's waiting for input to come in via the standard input stream. So if I type something, it reverses it. No, wait, that doesn't seem to work. Good, it does work. Line by line. And also, so I input a line, and I output a line, and then it immediately outputs, right? But if I were to say pass in my list of 10 names, it will read from that instead. Yeah? And similarly, I can redirect the output to a file, say, names rev.txt.
You see no more output to the screen, but it's redirected to the screen. You see no more output to the screen, but it's redirected to this file. And similarly, instead of redirecting to a file, we can pipe our output to another tool, which then reads from its standard input. And you know, that is the beauty of the command line, is this flexibility to solve a big puzzle, right, by combining all these pieces together in various ways.
And you know, that is the beauty of the command line, is this flexibility to solve a big puzzle, right, by combining all these pieces together in various ways.
And I would like you to, in a moment when we do the third exercise, I would like you to think about this as well. Yeah? Is your tool going to be really standalone? Is it going to try to do everything itself? Or is it going to be making use of other tools already available? And what are some of the tools that may want to consume its output? Yeah, that's another way to think about it. Yeah, I pretty much covered this.
Yeah. Has anyone ever tried reading and writing to the same file in one pipeline? No? Yeah? No? Well, you can't do it, so you shouldn't. If you do that, what happens is that the last process that is supposed to write to a file, to the file that you're also reading from, immediately opens it for writing, and thereby truncating the file. So the first process, which might be the same one, starts with an empty file, and as a result, you just delete your data. Yeah? But there is a solution. There is this, you can either write to a different file first, and then rename that, of course, or there is this tool called Sponge, which is not installed by default, but Homebrew has it, and it's also, you know, other package. You can very much, it's part of more utils.
Third exercise: thinking about streams
Yes. So, third exercise. Again, not a coding exercise, although you're welcome to code if you want to. I want you to think about these streams. Your command line tool, your next command line tool, it, I guess it operates on some data, right? It's what it often does. This data, is this always coming from, or do you really care where this data comes from? Yeah? Or would you be happy to just get some text, right? And then let the user, which may be yourself, figure out how to get that text, right? And then, let the user, which may be yourself, figure out how to get that text, whether that's a local file or something that you download, or maybe you use a command line tool to first extract some data from a database that then gets outputted to standard output that your tool can then read as standard input or from standard input, right? I would like you to think about that. So, we have three standard streams available to us. Which ones are you going to need?
You know, in some tools, the arguments and the streams, they sort of interact. As in, if you specify an input file, right, for a lot of tools that I've come across, if you specify an input file, it will not read from standard input. Yeah? There are tools that, if you don't specify anything, they just wait, right? Like we just saw with REV, or there are tools that just error. And the same holds for outputting, right? Are you happy to output to standard output so that you give other tools the flexibility to process that output even further? Or do you really want to write to a file, maybe because it's a binary format of some sort?
All right. I think that's enough time to think about streams. We're also halfway this tutorial, so I suggest that we take a quick break. You know, today's actually my birthday. Thank you. Thank you. Yes. It's been a hell of a ride. Now, to celebrate this, I brought Stroopwafels from the Netherlands. And I hope I have enough Stroopwafels. But if you would like one, you're very welcome to come over here and grab one. So let's take a five-minute break, stretch your legs, go to the restroom, whatever, eat a Stroopwafel, and then we'll reconvene.
Stroopwafel. That's right. Stroopwafel. Thank you. Thank you. Thank you. Yeah. I brought 72. So I was told that the maximum capacity was
So we'll see. We'll see how many are still here. Yeah. Of course. Yeah. Oh, that's good.
Because this is the process that we follow when planning, right? Yeah. My question is, do you, when you go, when you're planning your next CLI tool, you probably go through these thought exercises, right? Yeah, exactly. So is there any framework that you do to document it? Or you just open a note and start writing? Yeah, that's usually it. These tools are often very small. And they're mostly meant for yourself, to improve your own productivity, and to make life on the command line just a little bit more bearable. And, and so it's very much ad hoc.
And then there's always room for improvement later on. Yeah. Because sometimes you think about and then you start writing about how it should be. And then in my case, maybe you get engaged with something else. And then you go back to that after a few weeks. And then you start, okay, you retake your notes from that time and you start doing your code. Yeah. And then eventually that you need to take some, something that you have written to make it your own documentation for your future self. Yes. And that's why I generally advise against using ARGV, you know, this bare bones approach, so that you can easily, more easily add, so that you can make your tool self-documenting so that it has a help that you can then look into. Yeah, you're welcome.
Yeah, yeah, yeah, exactly. All right. Cool. Yeah, thank you. Have a stroopwafel, if you like.
Is it going well? That's good to hear. We're halfway. Yeah, we're sort of halfway with our slides. Man. Buttery smooth. Time flies.
Are more sessions coming before the next two, or just for one session?
I do just one session. One session, okay. Yeah, yeah, yeah, yeah, yeah. I mean, we just do one session. Okay. Let's walk around a little bit. Let's...
Happy birthday. Thank you. Thank you. Thank you. You're enjoying the show? Yeah?
Oh, good. And if you think they're general questions, then by all means. I mean, it's very likely that someone else has a similar question. So, yeah, always good.
Have you heard this? Do you know? That's what we're going to use.
Yeah, so does it mention PEP723? I think so. Or maybe not yet, because this was in December 2024. I'm not sure if that's it. Okay, yeah. It might be, but yes, definitely. Yeah, so we're on the bleeding edge. Yes. Hey, thank you.
Concepts vs. concrete steps
So, so far we've talked mainly about concepts. Yeah? And that's actually what I care most about. Syntax itself is not that exciting. Those are things you can look up or even let some LLM do for you. Now, I'm, you know, what I really want you to take home are the underlying concepts. But I also want you to know how to proceed. So, we're going to look at some concrete steps now in how to turn that very simple Python script, greet-simple.py, how we can turn that into a command line tool.
Now, I'm, you know, what I really want you to take home are the underlying concepts.
So, we're going to use very, very basic steps for this. Yeah? So, we're going to copy, copy the existing code. It's a bit, a bit, maybe, you know, obvious, but maybe you have your code in a notebook or a quarto markdown file or, you know, you need to just extract some portion of a larger code base. So, we're first going to make sure that we have the required code in a separate file. That's our very first step. Then, again, we're going to think about the arguments. Yeah? So, this is an exercise that you already did, and now we're also going to do this for this particular script. We're going to be thinking about, okay, the standard input. Or is there, are there other streams that we want to make use of? Then, as a fourth step, we're going to be looking at the very first line, the shebang line that our interpreted scripts could really use in order to start using them as a proper command line tool. I'll explain a little bit about, you know, making your code executable by changing the permission bits. And then, lastly, I'll show you how to add, you know, how to change your search path so that it really feels like a proper command line tool. Yeah? So, right now, I can imagine that you feel a little bit like Alice in Wonderland tumbling down the rabbit hole, but let's just take the red pill, and I'll show you how far this goes.
Okay. Yeah. So, this tool, which you can already look at if you have the GitHub repository open, that's our end state. Yeah? So, I'll walk you through the steps, leave this open, sort of as our to-do list. Yeah? Or maybe let's do this, the Great symbol. Yeah. All right.
Live walkthrough: turning a script into a CLI tool
Right. Yes. Let's just do this here. So, we have our starting position and the final position where we want to go to now open next to each other. So, you've already seen this one here on the left. That's our very basic script. It's not a command line tool yet because it's not executable. We still need Python. It doesn't take any arguments. It doesn't read from standard input. Right? So, those are the things that we're going to change. So, I guess where to start? Well, we already copied it. So, I would say that step one is now taken care of, argv. Where are we? Here. Yes, argv. Okay. So, we want to change the greeting. Yeah? So, let me just cheat a little bit here and copy from this other file.
And I guess we also need to import the sys module over here.
And then in order to make use of that greeting, I'll just do greeting like this.
So, now let's save this. Now, what we can do is, so, it's not yet a proper command line tool. You're right, but bear with me. Greet simple.py. And if I just, this is also how I ran it before. Ran it before. Now, it just, it still works the same, but I can also pass in one argument and then, hey, we have different output. So, we have now already added some flexibility to this script and, you know, command line tool in the making.
Right? Any questions about this particular step? Yeah? Okay, good.
The third step. Right. Our current tool reads from names.txt and there's, that's a fixed value. It's hard-coded. Yeah? So, we want this tool to take in, yeah, any data, right? Whether that comes from the internet or a database or a local file, we want to be greeting everybody.
So, instead of, I guess, I guess I need to, I closed, yes, all right. So, Python, or greet simple. Yeah, I shouldn't have closed that. Okay. But, so, instead of reading from names.txt, well, so, this is also a file. It's just a special file here in our Unix-like environment. And it's already open. So, we don't need to open this. It's already open. So, we can immediately start iterating. But for our purposes, we can just treat it as a file. Yeah?
So, this looks about right. Now, instead of closing this, let me,
you see, I just, I didn't pass in any file. So, it's now blocking. It's waiting for standard input. So, if I just say, hey, dice, just high dice. So, now, it behaves sort of what we also saw with rev, right? It reads line for line from standard input and immediately outputs the standard output. And they're both, you know, connected to the screen. But if I, right, if I send an interrupt, I just did control C. So, with that, you send an interrupt, and you get a warning. There are ways to make it exit nicely. But I can still now pass in names.txt using the smaller than sign. Yeah? Or I could,
oh, no, it has to be this. Yes. Right? So, it doesn't matter whether they're numbers or lines coming from a local file or some other process outputting something. Right? So, again, a lot more flexibility now that we make use of the standard input. Fantastic. So, that brings us to step number four. Oh, yeah, the shebang. You know, funny story. I once participated in a beatboxing competition. I did that once and only once. And I had to come up with a stage name. And I was, of course, already, you know, nerding out on the command line. So, I thought, like, shebang. Sounds awesome. Sounds powerful. Of course, when I was announced, it was pronounced as shebang.
So, the shebang, the name, the term comes from these two characters, the hash and the bang, the exclamation mark. Together, these are, you know, this particular combination are a particular set of bits that instruct the shell that the remainder of this line is the executable that should be interpreting the rest of the file. I hope I got that correct. Does that make sense? So, here we have, yeah, so, okay, so, there's a little bit of additional complexity here, because this is actually the executable that is going to be interpreting this. User bin env. And it takes one argument here, Python 3. So, all that this does, so, we can run this, we can say env, and we can say Python 3, and it runs Python 3 regardless of where the executable Python 3 is on my system.
So, using user bin env makes your tool more portable. Because it might be that on someone else's system, Python 3 is in a different location. So, this is not something that you want to hard code. So, this is very common. User bin env, Python 3.
So, using user bin env makes your tool more portable. Because it might be that on someone else's system, Python 3 is in a different location. So, this is not something that you want to hard code. So, this is very common. User bin env, Python 3.
Okay. So, and then later, we're going to look at a different line when we start using UV and PEP 723. Exciting stuff coming later. So, now that we have here, now that we have this, let's save this, we no longer have to specify Python 3. Yeah? We can, well, there's still a couple of steps that we need to do in order to make this feel like a true command line tool. Yeah? It still says command not found. We're going to solve that in a moment.
File permissions and the executable bit
Now it says permission denied. Okay. What's going on? This brings us to step number five. Every file and every directory in a Unix-like system has certain permissions associated with it. What is allowed? And currently, if we look at, let me see here. If we have a look here at the files in our directory, then in the very left, we see, yeah, rwx and some dashes. And these indicate the permission bits, the permissions that are associated with each file and each directory. Yeah? So, when we go to greet simple, that's the file we're currently working with, we don't see an X. So, we do not have the permission to execute this file. So, we need to change this. And to do that, we need, we use chamod or change mode. And there are some different ways to use this, but we're going to be setting, we're going to be turning the permission bit or the execution permission bit, the X, we're going to turn it on, which is the plus for the user. That's us. So, the user is, we are now going to set the execution bit to one, yeah, for greet simple.
And if we're then going to check again, you can see that the permission bit is now, or the execution bit is now set, because we do see an X here. And so, if we try to run this. So, previously, we got permission denied. I'm going to try again. It works. Right? And we noticed that we didn't need to specify Python 3, right, because of the shebang line. We're almost there. I would say that in a lot of cases, this is sufficient. Yeah? But that dot slash, what's up with that? Yeah? That doesn't really feel like a real command line tool.
Setting up the search path
We never have to use that. The reason is, is that this file is in a directory, which is not on our search path. Yeah? The search path is an environment variable, which is, that's a list of directories, where the shell will look for your executable, when you use its name. So, this is my current list of directories, right? Let me see if I can make this a little bit, can I do this? Yeah. A long list of directories, where it's going to look for. And it's in, it will stop the moment it has found an executable with the name that you used. Yeah? So, the order matters. Our current directory is not in here. So, we can do two things to fix this. We can either move our script to a directory that is in our search path. Yeah? And, or we can add the current directory to our path. Yeah? So, both work. What I really like is to have a, just one directory, somewhere in my home directory, maybe .bin or whatever, it doesn't matter what you call it, a place where you store all your own command line tools, and then just change your search path to add that one directory. So, it's a one-time setup, and then you're done and you just have all your tools in one place.
For now, for this tutorial, I'm going to do it a little bit different. I'm going to add the current directory to the search path. Where are we? We're over here. Yeah, so that's this. So, I'm actually updating the environment variable path to be a combination of the existing value of path and the current working directory. So, now, hopefully, fingers crossed, I can say greetsimple.py. Excellent. And I can even do this one directory up, although I still need the directory in front of here for the names.txt file. Tutorial slash, and it still works. Excellent. It works. Let's go back. And you know what? There's actually one more step that I like to do. We don't really need that extension. .py, that's not needed. That's not needed in Unix or Linux. So, we can just get rid of it. We can say greet simple.
Now, ladies and gentlemen, we can say greet simple. Hi.
Oh, of course, because we're in that, please don't laugh. Makes me nervous. No. Yeah, because we're now back again in our repository.
That's it. That's how you turn, even though it was just a simple script, how you can turn an existing Python script into a reusable command line tool with arguments, with streams, and whatnot. And it doesn't matter which approach you're going to use, whether that's the basic sys.argv, or as we'll see in a moment, the argpars module or typer, or what have you. These steps are always there. Are there any questions about these six or seven steps that I just performed in the back?
Q&A
Oh, you know what? That's a solution I never thought about. You can, of course, in your... So, the question was, is there any advantage to adding the script to a directory and put it on your search path than to just using an alias? And I think that's the solution I have never thought of before, but it's something you could do, is in your RC file, dash RC or t-shell RC, create an alias and then use the absolute path, the full path, the full location of that script. And is there a downside to that? I don't think so.
Although, you know, if you have... If you plan to create a couple of tools, right, then you would need to create an alias for every single tool. And if you were to just have them in the same directory, then it's only one directory that you have to add to your search path. So, that's perhaps the difference, yeah. You could also symlink, yeah. So, it's kind of like creating something extra into a directory that's already on your search path, yeah. But there's another, that's another solution. Oh man, I'm learning so much today. That's great. Any other questions?
That's right. Unless you're running this inside your Windows subsystem for Linux, or Docker image, or git bash, yeah. Yeah, yeah. Yeah, I know Windows is a wonderful operating system for playing games. Moving on.
And yes, yes, Linux is getting better and better. Yes. Thank you, Steam, for that. Sorry, I... Yes.
So, yeah. So, the question was, instead of a script, can we also run Jupyter notebooks from the canal lines? I am not sure. So, you would be invoking Jupyter something. There might be some, it might be some functionality that Jupyter then offers. So, you would be, if I were to do that, I would probably wrap that into a short bash script, or an alias, or a shell function, yeah, that would then call Jupyter. I don't know the actual invocation that you would need, but it might be possible. Now, when it comes to notebooks, this week I started playing with Marimo, a totally new way of working with notebooks. And these notebooks, they are
actual Python files. So, if you are looking for such an approach, where, you know, there is a notebook that you can also run from the canal line, then I can recommend Marimo. And I guess, if you search on YouTube for Marimo and Typer, then you'll most likely come across a video created by Vincent Wamadam, who works at Marimo, and he actually explains how to do this. It's pretty neat, pretty neat. Thank you. Oh, yes, and then there was one...
Handling arguments with argparse and Typer
Yeah, exactly. Okay, good question. So, in the tools that we just created, the tool, all the arguments are positional, whereas before I've been talking about short options and long options. What's up with that? Yeah, so that is something that argv, because it's so basic, it does not provide this functionality. If you want that, and you're still using argv, then you need to build this yourself. I wouldn't do that, if I were you. I would use either the argparse module or the Typer package. So, and that's what I wanted to show next. So, thank you very much for that question. I also see that we have about 10 minutes left.
Let's see what we can do here to make it fit. Right, so, so far, we've been using argv. Now, there is a built-in module, right? So, it's part of the Python standard library called argparse. And this gives you a ton of flexibility when it comes to defining your command line arguments.
It can automatically generate help. You can denote whether your arguments should be optional or required. It can do some post-processing. If you indicate, like, hey, this is a Boolean, you actually can use it as a Boolean instead of first parsing that string. Handles short and long options. Yeah. And it's, you know, let's have a quick look at what that might look like. Again, this is our greeting script, but now there's a lot more code. See here, these first, you know, lines over here, that's me setting up argparse, right? Instructing argparse. Okay, what arguments do we, does this script accept? Again, it uses argv under the hood, yeah? But it, yeah, it does a lot of work for you.
So, I recommend that you try this out. It's a lot more convenient. And if you have, yeah, more than, especially more than two arguments, I can definitely recommend, you know, moving on from argv and using argparse. Or use Typer, which is a relatively new Python package created by Sebastian Ramirez, the person behind FastAPI. This is a very modern way of declaring your command line interface, so to speak. Built on top of Click, which is, you know, and Click, you know, as we know, as I told you, it also uses argv under the hood.
The main difference is that with Typer, you use type hints. And from that, it generates your interface and the help associated with it. And it is generally a lot less code than if you were to use argparse. Let's have a quick look at that.
Again, it's our greeter application. And you can see here, I have a function main with some arguments, like functions have. But now, and these are, right.
Oh, right. So this one, I should update the repo. There's now a better way of doing this. Sorry, don't worry. But what I want you to focus on right now is, it's not so much the code itself, but just, you know, the takeaway here is that Typer uses type hints. And this function is still usable as a regular Python function. And that is not the case when you use Click. Click really changes the signature of the function. So that is one advantage of Typer. See, so that's a lot less code than if you were to use argparse. Now, that's Typer. But what is this here at the top?
Inline dependencies with PEP 723 and uv
This is super awesome. This here is recently approved PEP 723, which allows you to specify dependencies inside the script itself, right? So the syntax, you know, is perhaps a little bit weird. You have to start with three slashes, script, and then it's something that looks like pyproject.toml. You know, you'll get used to it. And uv, you know, this new tool to handle your packages can help with this. I have links in the presentation at the very end. But this allows you to make your tool standalone. And this is definitely a good thing when you use external dependencies. You can specify the Python version that your tool needs and any other external dependencies go in here, in between these brackets. And there is at least one, which is Typer itself. But also if your tool uses, say, the requests library or pollers, yeah, it works. And if you then, because we now use a slightly different hashbang line, so it's no longer Python, which runs this code. No, it's uvrun. It detects whether you have these packages, these dependencies already installed. And if not, it will do so. Making it, making, yeah, the ability to share this script with others all the more easy. Let me see if I can quickly demonstrate this.
This here is recently approved PEP 723, which allows you to specify dependencies inside the script itself, right?
Great. Typer. Oh, yeah, let me first show that. Oh, that's probably because my path in this particular session was not updated. Wow, look at this. It's beautiful. That's the help auto-generated by Typer. And normally, if you would run this script, this tool, for the very first time, you will see that it installs some packages. Now, this was not shown here because I already had these packages installed because I already ran it. So it's being run in its own virtual environment. Yeah? These packages are not installed globally. This process gets its own virtual environment. Pretty cool. And I think in the repo, I've included two examples that are, yeah, a bit bigger, or three, actually. There's the analyze tool, both an argparse version and a Typer version. There's also sample.py, which is something that I created a long time ago that I actually needed. It's being used in the book Data Science at the command line, just to give you another example of how a more elaborate command line tool may look like. Yeah? So these two analyze scripts, they also use subcommands.
Wrapping up
You can have a look at these scripts yourself. Yeah, there's some best practices for you to review. I don't have time now to go over all of these, although some of them have been mentioned along the way, some anti-patterns as well that you want to avoid. And so this is your take-home exercise, right? It's to start building, and preferably as soon as possible, because otherwise, you know, you will have forgotten all about this. And if you have built something, I would love to hear from you. If you have any other questions after this tutorial, you know, I want to hear from you. So with that, I want to thank you very much for attending this tutorial. I think you've made the right choice, right? These 90 minutes, they're going to benefit you for the rest of your career. Yeah? So again, thank you very much. Enjoy the rest of the conference.
