image: thumbnail.jpg
Hadley Wickham | testthat 3.0.0 | RStudio (2020)
testthat 3e also introduces the idea of snapshot tests which record expected value in external files, rather than in code. This makes them particularly well suited to testing user output and complex objects. I'll show off the main advantages of snapshot testing, and why it's better than our previous approaches of verify_output() and expect_known_output().
Finally, I'll go over a bunch of smaller quality-of-life improvements, including tweaks to test reporting and improvements to expect_error(), expect_warning() and expect_message().
Webinar materials: https://rstudio.com/resources/webinars/testthat-3/
About Hadley:
Hadley Wickham is the Chief Scientist at RStudio, a member of the R Foundation, and Adjunct Professor at Stanford University and the University of Auckland. He builds tools (both computational and cognitive) to make data science easier, faster, and more fun. You may be familiar with his packages for data science (the tidyverse: including ggplot2, dplyr, tidyr, purrr, and readr) and principled software development (roxygen2, testthat, devtools, pkgdown). Much of the material for the course is drawn from two of his existing books, Advanced R and R Packages, but the course also includes a lot of new material that will eventually become a book called "Tidy tools".
Transcript#
This transcript was generated automatically and may contain errors.
So today I am excited to talk about the 3rd edition of testthat. So I'm going to start by talking a little bit about what do I mean by that. You certainly want to know what a version of a package is, but what do I mean by this being the 3rd edition of testthat.
Then I'm going to talk through three big new features in this version, in this edition, see how testthat now displays differences much more clearly using the waldo package. We'll talk about a new type of testing, a new type of testing to testthat called snapshot tests. Very briefly talk about testing in parallel, multi-process testing, and then finally I'm going to give you a real quick goody bag of a bunch of little small stuff that will hopefully make your life a little bit easier when using testthat.
Introducing the 3rd edition
So the big news with this release of testthat, this upcoming release, testthat 3.0, is we're going to introduce the idea of this 3rd edition. And I should say anything you're going to hear about today is also explained in more detail in vignettes on the testthat website, so if you want to learn more about anything I say today, feel free to take a look at those vignettes, and if they don't explain, they don't help you understand what's going on, please file an issue so we can learn better.
So the basic idea of the addition is that we want to make breaking changes to testthat. Testthat's like 10 years old or something, and there's a bunch of things in the package that we now regret. But testthat's kind of a victim of its own popularity. There's something like 4,000 or 5,000 packages on CRAN that use testthat, and many of them are not actively developed. So there's no way we can change testthat without breaking a bunch of packages, and many of these packages, since they wouldn't be maintained, they'd just end up off CRAN because people wouldn't have the energy to fix them.
So the idea of an addition is that this is something you have to deliberately opt into. So if you want to use many of the new features or all of the new features of testthat I talk about today, you have to deliberately choose to use a 3rd edition of testthat. And to sort of do that, you basically have to add a line in your description file, config slash testthat slash addition. So if you don't do this, you'll continue to use the second edition, that's the existing behavior of testthat. If you do this, you'll get, if you do change this, you'll get a bunch of new features and a bunch of things, which I'll explain shortly, which have gone away.
So the kind of idea is that it should take hopefully, you know, less than 30 minutes to convert a package to use the 3rd edition. You know, it's a little bit of work, hopefully not too much work, but it just generally gets you up to date with all of our best current best practices. And I should say, if this kind of idea of this addition idea is successful, it's something we're likely to try out in other packages, a way of allowing existing code to continue working while giving you the choice to opt into a new set of behaviors.
So additions will always be coupled with a major version. So that means testthat2 and everything else before this counts as the 2nd edition, testthat3.0 can use the 2nd edition, that's the default behavior, or you can opt into using the 3rd edition. And if one day, you know, many, probably multiple years down the line, we decide to introduce a 4th edition, then that would be tied to testthat4.0.
What's going away in the 3rd edition
So what's going away in the 3rd edition? These are the things that are most likely to lead you to change your tests. So while the context function is going away, we've kind of been moving away from this for quite some time in favor of just using the name of the test file. There's no need to kind of duplicate the information in the test file and another function, which you have to remember to update if you ever move your test files. We've also been lately in devtools and use this building kind of a stronger coupling between files in the R directory and files in the test directory. So that is a kind of a one-to-one correspondence, which gives you a bunch of handy keyboard shortcuts and development tools.
The very old expect that function is going away. This was the kind of first API, the first excessively clever API that testthat used with every test kind of had the flavor. You could read it like a sentence like testthat blah is true. And I pretty quickly decided that that was excessively clever, not very good. And so we're finally deprecating that for good.
Expect is we're deprecating in favor of expect S3 class, expect S4 class and expect type of just to be more precise about what exactly you're trying to express. Expect equivalent. I'll talk about a little bit later, but we're basically that's just going to be an argument to expect equal or expect identical. Some of the mocking functions going away in favor of more featureful packages. And then finally, we are moving away from set up and tear down in favor of test fixtures, which I'm not going to talk about today, but there's a whole vignette about them if you want to learn more.
And hopefully that's kind of all I need to say. Just these are the functions that are going away. Like if you really, really believe any of these functions are important, like now's the time to speak up and say you want to keep these. Otherwise, when you switch to the third edition, rerunning your tests, you'll get deprecation messages telling you what to replace these functions with.
Clearer comparisons with waldo
So that's kind of the stuff that's going away. Now I want to focus on the cool new features. I think the feature you are most likely to encounter and the most likely to give you pleasure in your life is that comparisons with expect equal and expect identical or test failures with expect equal and expect identical are now much easier to understand.
So previously expect equal uses the base all.equal function. This was never really what all.equal was intended for, but it kind of did an okay job. But there's a few cases where it just doesn't give very useful output. And so here I'm comparing empty cars without the first column to empty cars, the full data set. And you'll notice I get a lot of output. What tells me that there's 10 string mismatches in the names, that doesn't really help me. It tells me there's a length mismatch and then it gives me a bunch of mean relative differences. So this certainly tells me that there's a difference, but it doesn't kind of concisely point me to the fact there's a column missing in this data frame.
So now we use waldo. So to use this, I'm just going to use this function called local edition. This is not something you generally need to use in your tests because you'll convert a whole package to use a third edition. But we provide this local edition function for writing stuff like this, for writing vignettes so you can explain about this new edition. This just temporarily sets the edition to the third edition.
So now we use the third edition and now this is the new waldo output. So the kind of the first principle is that waldo is going to give you the most important differences first. And in this case, I think the most important difference is that the lengths are different. And then it tells us what the names are, how the names have changed, lining up the old names, lining up the actual names with the expected names. And then it uses color to highlight the fact that in the expected, there's another column called MPG. And then finally down here, we get an exact description that, well, the names are different, but the values are also different. The actual, it's absent.
And as I read this, that is clearly not the case because it's absent from the expected. So I think I've messed up the labels here. So I'll have to fix these. This should say expected MPG is absent, and this should say actual MPG is a double vector. So that just allows you, once I fix those labels, to precisely see what the differences are.
So here I've switched to use the third edition. Waldo then shows the most important differences first. It uses color to help highlight differences, and it always uses the names of things we have possible, just because that makes it easier to understand exactly where the differences lie.
Waldo then shows the most important differences first. It uses color to help highlight differences, and it always uses the names of things we have possible, just because that makes it easier to understand exactly where the differences lie.
So another little example here, I have a factor. This is an ordered factor with one extra level. When you look at the expect equal output from the second edition, you'll see like it kind of tells you that they're different, but it's hard because it doesn't include the values to know exactly what's changed. When we switch to the third edition, now we can see, well, there's a new element of the class vector. It's ordered, and there's a new element of the levels, which is D, and hopefully this will make it much, much easier to see exactly what's gone wrong. And certainly in my own, this is like, this is the number one reason I've been converting packages to use the third edition to test that, just because it makes things so much easier to see when something's gone wrong.
So if you want to learn more about waldo, you can go to the waldo website, which shows a bunch of kind of examples, shows some of the principles. A lot of the differences of the values are powered by this really nice package called diffopt by Brody Graslam, which uses the same algorithm as the diff utility on Linux, which just makes it really easy to kind of narrow in exactly what has changed between two vectors.
This change also makes it now possibly precise about what's the difference between expect equal, expect identical, and expect equivalent, which are always a little vague in the past. Now all of these functions are equivalent to expect identical, but with some extra arguments set. So if you use expect equal, that's equivalent to expect identical, except that it ignores small floating point differences. Or if you use expect equivalent, which is now deprecated, it's just the same as using expect equal or expect equivalent with ignore attribute equals true. That's the only thing that expect equivalent does is it just ignores attributes. Now we're going to make that precise in a function argument. And then any other arguments in expect identical or expect equal are passed on to waldo compare, which gives you the ability to kind of fine tune your comparisons as needed.
Now this is such a great idea. You might wonder, well, why can't it work for the second edition as well? Well, unfortunately, when I implemented expect equal, it turns out that I made a rather silly mistake when implementing the tolerance comparison. So depending on exactly which code path it goes down, it computes the tolerance in slightly different ways. One way it always uses the absolute tolerance, and one way it uses absolute or relative tolerance. So there's basically no way I did a few experiments. There's no way, regardless of which one I pick, it causes like hundreds of CRAN packages to break. So that's the main reason that this has to be in the third edition.
But there's a few other small things, particularly if you're testing code in a modeling package. All.equal, which used to power expect equal, ignores environments of formulas and models. Sorry, formulas and functions. Waldo does not, but it provides a couple of arguments to allow you to override that if you need to. And there's just a few other small things around like POSIX CT. So I think the behavior of waldo is generally better. It's a little bit stricter, and when it fails, it'll give you clearer results. But we can't just turn it on for everyone because it would break a bunch of tests. So you're going to have to deliberately opt into it.
Snapshot testing
Okay, so that's waldo. I think that's like kind of the number one feature, like, again, I have found it like so useful when debugging tests myself to have that much, much, much clearer display. The next big feature I want to talk about is snapshot testing. Again, this comes with a vignette, so if you want to learn more about it, you can read the vignette. If the vignette doesn't make sense, please file an issue so we can make it better.
But the basic idea is this is a new type of testing. So normally in unit tests, you describe the expected output using code. And in the vast majority of cases, this is a really, really good idea because it allows you to kind of describe what you expect. And in some sense, the tests help document, because the tests are code and code is a means of communication. They kind of help describe the expected behavior of the functions as well. But sometimes describing the expected output is just like really annoying, like if it contains a bunch of special characters, like quotes or backslashes, you have to spend a bunch of time like carefully escaping them. And then when something goes wrong, it's hard to see exactly, like you've got to unescape them in your head, and it's just a pain. Or maybe it's very large, like maybe you want to check an entire HTML page or multiple paragraphs of text, as you expect. Or maybe it's not even something that you can easily describe with text. It's an image. Or something else.
So the idea of a snapshot or golden test, which idea used in other programming languages and other testing packages, and test that inspiration draws primarily from Jest, which is a JavaScript testing package that Joe Chang shared his experiences with me a bunch and persuaded me that this was something really useful. But so the snapshot test, the key idea is that instead of recording the results inline in the test itself, they're stored in a separate file. And testthat provides a bunch of tools for looking after that, for managing that file. So it will create it automatically the first time you run it, and then it gives you tools to update it when you decide there really has to be a change. And if you've used verify output or expect known output, which we've never really advertised because we've never really been particularly happy with them, the snapshot tests basically
Snapshot tests
So what does a snapshot test look like? So here I'm just going to give you a quick simulation in the presentation, and I'll show you what this looks like in a real package. So here I've got two files, foo.r, I've just got this very silly and simple function, which has a mistake in it, which you can see, which we'll fix shortly, and then I've got a test. So I run foo, I expect it as a character vector, and then I'm going to use this new expectation, expect snapshot output, and this doesn't have what the expected value is, because it's going to save it to test.
So the first time I run this, it's going to say, warning, create snapshot reference, and it creates a new file, and I forgot to put the correct directory in here, inside the test that directory, it's going to create a new directory called snaps, underscore snaps, and inside that, it's going to have a file called foo.md, right? So our R file is called foo.r, our test file is called test-foo.r, and then our snapshot for that, the snapshot file is going to be called foo.md. So all of these are just named, have a strong naming convention, so you can easily find which snapshot corresponds to which test, which corresponds to which R file.
So what does that snapshot contain? Well, it's a markdown file, I'll kind of explain the syntax shortly, but we use a heading to indicate that this is the test, and then it's going to put the output of that test directly in that file. So now if I run it again, if I run that test again, the test is going to work, because nothing's changed, and so the test will pass.
If I change it to fix that typo, I run the test again, I'm getting an error, saying that the previous value was something conflicted, and the new value is something complicated. Now the downside of snapshot tests is that there's no way to know what is correct, so you as a human now have to step in and intervene. So if this is correct, if you really did mean to make this change, you can run snapshot accept, and that will accept the change. And the way that this works is that when you have this value, when something is changed, there will be a new markdown file, which contains the new value. If you accept it, then it will replace that with the old value.
Okay, so that's kind of the big picture. So let's just dive into that for a slightly more realistic example. Start with a simple one. So I have this package, I have the same test which I copied before, and I'm going to run this test. And I'm going to run this test by pressing commands T, which is a shortcut for devtools test file. So this keyboard shortcut takes advantage of that convention, that if you've got a file called foo.r, the corresponding test will be test.foo.
And so this says I've added a new snapshot, this is the value of that snapshot, I can look at it, there's this markdown file again, which I'll explain shortly. And then if I run that test again, you'll see that all the tests have passed. The other thing, if you've used devtools test file before, which I'll talk about briefly later, is there's now a slightly different display, a slightly more compact display to hopefully make it easier to run tests for a single file interactively.
So I'm going to change this, I'm going to correct that typo, I'm going to test again, and now the test fails because the snapshot has changed. So it shows you the current value, which is something complicated, and it shows you the previous value, which is something complicated. Now, if this is a deliberate change, you can run snapshot accept. If this wasn't a deliberate change, oh, that was just a typo, I can fix it and rerun and all my tests pass again.
So if you've ever used verify output, this is a little bit different because verify output automatically updates the kind of true known value on disk, which forced you to use it with GIT, basically. So I'm going to fix this, the test fails, you can see now in my snaps directory, I've got foo, that's the foo.md, that's the previous value, and I've got foo.new.md, which is the new value. So I look at this, I decide that is actually correct, and then I can just copy and paste this, and now foo.md has the correct result, and I can test that again.
Okay, so let's look at a slightly more complicated example. So this is a bullets function, and it's basically used to create HTML bullets. So it looks like this. And this is like a little annoying to test, because if you're going to put this in a test, you'd have to escape all of these new lines and kind of carefully manage all the white spaces, it's just a pain. To test this code, it's a little easier to use a snapshot test.
So I can run this test, and now here is a test snapshot file with multiple expectations in it. The first one, I am just using a bullet with a single bullet, and the second also has a single bullet and also sets the ID. So if I later want to change this bullets function, maybe I've decided actually I don't want this indent. Let's get rid of that. I rerun the test. Again, the changes use waldo to highlight what's changed. So it's a little bit hard to see this, but this is the thing that's the same. It's in gray. What's changed? We can see there is now no space where there was before.
So I look at this, I decide, yep, that's a deliberate change, and then I run snapshot except bullets to update that snapshot. And that basically is snapshot testing.
Of course, there's more documentation, and I think it will take a little while to get your head around it all. I've shown you expect snapshot output here. There's also expect snapshot, which, as well as printed output, also captures messages, warnings, errors. There's also expect snapshot error if you want to capture specifically just error messages. And then we've got expect snapshot value, which captures return value. So this is a little bit different. This is if you, for example, wanted to test the output from a complicated function, you just want to make sure it doesn't change without warning. You can use expect snapshot value.
And then again, of course, there's the vignette, which you can look at. Well, I guess I can't show it because I've installed the development version of the package which doesn't have the vignettes. But you can go to the website. Actually, let's just go to the website. And then we can see all these vignettes. Here's one on snapshot testing. I'm just giving you a little bit more detail about what I just talked about.
Snapshot file format
So the only other thing to mention is just what do these files look like? Why are they markdown files? Well, if you're writing this, this package is a collaborative project and you're using GitHub or some other tool where you do code review, it's really important to make sure it's really important that these snapshots be human readable because when someone goes to review your code, they need to go look at the snapshot and say, is this a reasonable change or not? So they use markdown. Each file, again, is there's one snapshot file per test file, which normally corresponds to our file in the R directory. There's a heading for the test name. And then if you have multiple snapshot expectations and a single test, they're separated by the horizontal row with three dashes.
it's really important that these snapshots be human readable because when someone goes to review your code, they need to go look at the snapshot and say, is this a reasonable change or not?
One thing that I'm working on at the moment and some help from Joshua Kunst is a Shiny app that will help you review all of these differences and accept them or reject them with by clicking buttons rather than typing stuff at the console. And the other big part of that is providing tools for doing image snapshots as well. So this is something that we've implemented in two places and Shiny test, which is used for testing Shiny and VDiffer, which is used for testing ggpod2. The idea is they're going to pull out the common code, centralize and test that and invest a bunch in this whole snapshotting idea. So you've got a really nice workflow if you do need to do image tests. Image tests are complicated because they can change for all sorts of reasons unrelated to your code. But sometimes they're all you have and they are a really important part of testing both Shiny itself and ggpod2 itself.
Okay, so that's snapshot tests. Main idea of snapshot tests is that compared to regular unit tests, which have the expected results in the test file, snapshot tests store the expected results in another file, which makes them suitable for testing large output, output with quote marks and backslashes in it and for testing things that you simply cannot describe with text like images.
Parallel tests
Next, I wanted to talk briefly about parallel tests. This is still work in progress by Gabor Chadi. And again, you're going to have to activate this specifically for your package again by putting another line in your description, config, test that parallel true. But the payoff for this is pretty big. It's going to run your tests on multiple processes. So if you have long running tests, this is going to make a big improvement to the total running time of your tests.
Now, the downside is like there's a little bit of overhead associated with that starting out multiple R packages, multiple R processes, loading all the code in those. So there's a little bit of overhead. So if your tests are very fast, like if your tests all run in under a second, it's probably not going to have a huge impact. But if you've got tests that take five or ten seconds, the whole test suite takes minutes, then this should hopefully have a really big positive impact on your workflow.
There are some downsides. As well as this big upside of speed, tests will now kind of effectively run in stochastic order because you'll have like four processes and each process takes kind of the next test in the queue. And so depending on exactly how long each test takes to run, the tests might get run in a different order. And this basically means if you have any dependency between your tests, which is relatively easy to introduce because normally they will run in alphabetical order. If there is any dependency between your test files, you'll get like random test failures that occur sometimes but don't occur other times depending on exactly which order the tests are run in. In other words, this is like a dependency debugging nightmare.
So we're still thinking about tools to kind of ensure that if that happens to you, we've got some mode debugging mode you can switch on to get more insight. And then also, if you have used like global setup or teardown, for example, like you set up a database or some CSV file for all of your tests, you're going to need to think that through in a little bit more detail because those setup and teardown files are now run by multiple processes. So there's going to be a little bit of work to convert your tests to use Parallel. We're still working on this. You know, there's a vignette if you want to learn more and want to try it out. But we're hopeful that this will make a big impact if you've got long running tests.
Summary and other new features
So I've talked about this idea of the third edition. This is a special mode. You'll have to switch on if you want to use all the latest and greatest test that features. We've talked about waldo, which makes comparisons or test makes test failures from expect equal and friends much, much easier. We talked about snapshotting tests. We talked about running tests in Parallel. Now I just want to show you kind of a bunch of little features that I think are kind of cool.
So in the course of working on test that for this release made a bunch of improvements to the reporters. The reporters are the things that actually go and generate the results. So when I press test, this is done by a reporter. If I press command shift T to run all of the tests, these are different reporters. One reporter that you never normally use is called stop. You don't normally call it explicitly, but it's called stop reporter. That's what happens. And that's the test that's run. So that's the reporter that's run when you run a test interactively.
So here I'm just running a test, which is not working for probably reasons that I need to look into. But if I create a share of another example over here, if I run this test, this is the stop reporter. Now it clearly tells you that your test passed. And it gives you some emoji. If your test does not pass, it nicely displays the failures. And it also displays any warnings. And really conveniently, it also displays the backtrace for the warning.
So if you've got warnings in any of your tests now, they get a full backtrace. You can figure out exactly where that warning came from.
One of my kind of pet peeves is, let me do this, is this. I have partial matching warnings turned on. And if this occurs somewhere deep inside a function, inside a function, inside another function, it's really hard to track down. Now in your test, you get this nice backtrace so you can figure out exactly what sequence of functions leads to that problem.
OK. So that's the stop reporter, which is used in debugging. So now uses color, uses emoji, and generally gives you more information about problems when you're interactively running tests.
I showed you earlier that compact progress reporter. That is the single line here, which gives you kind of running. This is a little bit lower, a little bit fast. We could do slow test to expect true system. So just doing a little loop here, which is going to do an expectation, and then sleep for a little bit. So you can see it counts up a little slower. So you get like a running progress bar in some sense of all the tests that are running so you know exactly what's going on.
Reporter improvements
I've made the regular reporter, which you see most often when you're running all the tests in your package. The biggest thing is I have added a bunch of new praise that uses more emoji, because I think emoji are fun. These random praise kind of veer into a little bit into dad joke territory. But hopefully that's just something, a fun little feature of testthat that keeps you motivated and keeps you going when your tests aren't working well.
And finally, the last reporter is the check reporter that's run inside of our command check. Now it reports all of the problems. So it also reports warnings and skips tests. It tells you all of your skips tests by type, which is really useful for checking that you haven't accidentally skipped tests that you meant to run. And it also creates an RDS file with a machine readable list of all of the tests. That's probably not something you're going to use directly, but it is something that we will start to build into our tooling so that things like GitHub actions can give you nicer displays of your tests and so on.
Changes to condition handling
We've also made a few changes to the way that the condition functions work. So here I have a function that calls warn and message. And now if a condition like a message, a warning, or an error is not explicitly caught by your expectation, it will continue to bubble up.
So if I run this line, if I run this line, this expectation passes. It does find a warning called hi, but the message called bye still bubbles up. Or if I say I expect a message called bye, I'll see a warning called hi. So if I want to capture both of them, I need to use expect message and expect warning together. This hopefully will give you better control over exactly what's going on. It's going to cause a few more warnings in your tests.
These won't cause your package to fail our commands check, but they will require a little bit of work to make sure that those tests are as you expect. If you want to just ignore them, you can still, rather than using expectations, you can just switch to the base functions, suppress messages and suppress warnings, if you don't care about them.
Now, you might notice here that we've got expectations nested inside of expectations, and you might kind of naturally think, well, why can't I use the pipe for that? Unfortunately, you cannot currently do that because the pipe eagerly evaluates everything. So if I do this, you get a bye and a hi, and then this fails because this is called before this is called. We will start to announce a new work in Magruder that will make this work and hopefully make Magruder a little more compatible with the native pipe that is unlikely to appear in the next version of R as well.
Okay, so that also means if you're one of the, I don't know, 10 people in the world who use the all arguments, so expect warning, expect message, that's now deprecated. You have to take a slightly different approach, but I'm pretty confident that's going to be a much nicer API overall and shouldn't change existing behavior too much.
And the last thing is if you've ever used expect error and it gives you this message to set the class, we no longer encourage you to do that. It kind of fixed one type of test fragility, basically the cost of just introducing a different type of fragility, and so now you're better off just using expect snapshot error if you want to check that a specific error occurs because that just gives you a bunch of nice features for managing the change over time.
Wrapping up and what's new in testthat 3.0
Okay, so I'm just about to wrap up, which is great. So there's plenty of time for questions. So what have I talked about today? First, testthat 3.0 is coming out soon. I should say we'll probably start the release process in about a month, which means it's at least two months before it's on CRAN. So it would really be great if you would try it out, and this is the right time to let us know if this is causing you pain so we can fix it before release. Plenty of time to do that.
So the third edition is going to be part of testthat 3.0, which will be on CRAN and at least, well, the soonest possible is probably two months. It's likely to be a little longer than that. The third edition you will have to deliberately opt into. That gives you a bunch of new features at the cost of doing a little bit of work to clean up old APIs. The new edition uses waldo to make comparisons, which will hopefully make your test failures much easier to debug. It provides snapshot tests, which are an alternative form of testing where the expected results are stored on separate files rather than inline and code, and then the testthat provides some functions to help manage those files so that you can accept changes when you have made them deliberately or revert changes when you've made them accidentally.
The new edition uses waldo to make comparisons, which will hopefully make your test failures much easier to debug.
testthat 3.0 will also introduce parallel testing, which, again, will be a little bit of setup work just to make sure your global setup and teardown works appropriately and you don't accidentally have any dependencies between your tests. But the payoff will be if you have slow tests, they should run much, much faster because they'll run in parallel. And then, finally, I showed you a bunch of goodies, many of them featuring an emoji that will hopefully make your day-to-day life using testthat a little bit more fun, a little bit more pleasant.
So if you do want to try it out today, you'll need to get the development version of testthat and the development version of devtools. And then in your package, if this is a package that uses continuous integration or similar, you'll need to add testthat to remotes. And you'll need to accept the addition to the third edition to take advantage of all the latest and greatest features.
So that was a lot of content in 40 minutes. Again, hopefully, if you didn't take in everything I was saying, there's plenty of additional material and all of the vignettes. And if you don't find those understandable, please let me know so we can make them better. Thank you. And now, hopefully, Jenny will have some questions for me.
Q&A
Yeah. So the first big one is I think people want to hear more about why you're doing a third edition. So why not create an entirely new package the same way that plyr led to dplyr? Or why not create testthat 3? Or why can you not do this through semantic versioning of the existing package?
Okay. So the first question is, like, why not a new package? And that was something I considered. But, like, 90% of the code between the second edition and the third edition are the same. Like, the biggest difference with the third edition is it just takes stuff away from you, stuff that we now regret. So if we created a new package, we'd have two packages that have, like, 90% overlap in the code. And whenever there was a bug, we'd have to remember to fix it in both packages. So I think for this case, where, like, most of the stuff is the same, we're just trying to get rid of some new things, I think creating a new package would be overkill.
The question about semantic versioning is something that, like, testthat does use semantic versioning. We testthat 3.0. But that doesn't really help in the R community. Because you only have a single version of a package installed on your computer and a library. You can kind of work around that. But it's a little fiddly. And for packages on CRAN, they always use the latest version of the package on CRAN. So if I just released testthat 3.0, with all of these new features in it, like, something like 500 packages would break on CRAN, because they would automatically use the version of testthat on CRAN. So I think that's why those are, like, we want to make a little bit of friction, so that you have to deliberately choose to use this new testing, use these new conventions, use this, basically stop using old stuff that you should have stopped using a while ago, without duplicating a bunch of code that we'd then have to maintain in two places.
Okay. So now I have a series of questions, and the list is growing longer, that are smaller. And we'll just work through as many as we can. So this first one is, have you tested whether snapshotting plays nice with things like cover or the good practice package? Does this third edition work have any effect on Shiny test? So kind of talking about how this works with other packages.
Yeah. So cover, I mean, snapshot testing is just testing. It just works exactly the way you'd expect with cover. I don't know that much about good practice. I would think it would not, I don't think it would cause problems for good practice. Shiny test, it's not clear precisely how this will play out. I've been having kind of bigger discussions with the Shiny team about how testing Shiny should work. Because my vision for testing Shiny is quite different from their vision of testing Shiny apps. And so we're just working towards that. I think in an ideal world, eventually Shiny test would use kind of more of this new snapshotting infrastructure that testthat provides. But there's some other big changes that have to happen to Shiny test. So it's hard to say today whether the Shiny test will eventually kind of integrate this better or there'll be a new package that's kind of a successor to Shiny test that uses some other stuff and is more focused on snapshot testing by testthat.
If you want to kind of see my take on testing Shiny, I'm working on this chapter at the moment. If you follow the Shiny test repo, you'll notice that I've recently triaged like a hundred and something issues. So I'm thinking about this a lot. I'm starting to work on it, thinking about the test. I don't exactly know what the answer is, but you can kind of see my latest thinking in this package, in this chapter of mastering Shiny, which will get updated pretty rapidly in the next week or two.
All right. The next one is, do test snapshots need to be included in the package or could you exclude them if they're large and then developers can recreate them from a stable version? I think if you want people to be able to run your tests, you have to include them in your package. Because they are literally like the correct output. One thing I did forget to mention is that snapshot tests are not right, will be skipped on CRAN automatically. You can opt out of that with an argument if you want. Just because if a snapshot test fails, it isn't necessarily true that it's like a real meaningful failure. It might be an incidental difference or a difference from a downstream package that you don't want to cause a failure on CRAN. But yes, if you want people to run your tests, you have to include the expected results in the package.
All right. Is it possible to use snapshot tests if the function does not return anything, but rather has a side effect? So I mentioned briefly expect snapshot, which captures the side effects of messages, warnings and errors. If it's making other changes to the global state, you would need to make that kind of
explicit. You know, like if it's attaching, if your function attaches a package, you know, you might do expect snapshots, like search, which would capture the search path.
Or maybe if it's creating files in the directory, you could use, so it doesn't directly handle side effects apart from messages, warnings and errors, but you could easily wrap up a little function that captures those side effects and makes them explicit and snapshot that.
Context deprecation and JUnit reporter
All right. The next person missed a little bit about what's going on with context, but sees that it's deprecated and thinks that it's needed when using the JUnit reporter. So how are those two things going to work out?
I don't, so let me just find a, let me see if I can quickly find a package just so I can explain the context for this change to context a little more.
Okay. So this is just a per package. I just searched to find all the uses of context and you can see like the test-lmap.r file contains a called a context lmap. The test-mapper.r calls as it contains a called a context as underscore mapper, which is a little mildly inconsistent or test-win contains a called a context win. Test-utils contains a called a context utils. So by and large, it's like a one-to-one mapping between what you put in the context and the name of the file.
And that is just, you know, you don't want that duplication because you end up with inconsistencies. Well, maybe if I can find, well, here we can see that this is a test for pmap. It probably used to be test-pmap, but we renamed the file and forgot to update the test context. And so now there's like this mismatch between the two. So that's the reason why we're getting rid of context because it's just, it just introduces this duplication from little real guys.
But the context is still kind of implicit. Like we just take the file name. So the context that tests the JUnit reporter, for example, we'll use, we'll just be map underscore in. So it shouldn't change anything practically with the JUnit reporter.
Mocking functions in external packages
Okay. So this is a person like me who uses with mock and local mock. So if they're going to be deprecated, what is the suggested way to mock functions in external packages?
I read that the goal of mocker, the package mocker, is to provide a drop-in replacement, but it does not have this feature. So basically, the way that with mock works is truly an abuse of R, and it does something that I now believe to be tremendously ill-advised and the chances of it stopping working in a future version of R are high, because what it does is so, you know, it's like, you know, future version of R are high, because what it does is so bizarre and horrific.
So basically, you have to use another approach. So both the mock R and mockery packages provide slightly different techniques. Unfortunately, you just have to use those.
Although I'm just so concerned that the way that with mock works is not, it's not good, and it's surprising that it hasn't caused any problems to date. So I just, yeah, unfortunately, it's like something that is really nice and really useful, but was achieved in a really, really horrible way.
Snapshot testing: size limits and mixing with regular tests
Um, does Snapshot have a limit on the size of the data? And can you combine Snapshot tests with more explicit tests of the Snapshot?
Uh, yeah, so that was kind of, so there's no limit on the size. And I think, you know, generally, you know, you'll, you'll, you will sort of mingle Snapshot testing with regular testing, like where you can, like, if this is a list, you might want to test that this is a specific value. And there's another component of the list that's just a, you know, a big blob that you don't want to kind of type out. So I would imagine, you know, I really imagine that you will want to mingle those as much as possible.
I think it will be tempting to overuse Snapshot testing, just because it's so convenient, because you have to write like the right hand side of expect equal, you have to, you know, to make clear in your head exactly what you're expecting. But I think if you do that, if you, if you use Snapshot too much, you'll find your tests are a little bit fragile and kind of break more often than you're otherwise like, and that will kind of push you naturally back towards a finer grained unit testing that you're, that you're used to and test that.
You know, of course, putting really large files in your package is going to be annoying for other reasons. So I don't think you want to put like, you know, megabytes of data using this, but certainly like kilobytes of data, which would be a pain to put in a test file. It's a really nice sweet spot for Snapshot testing.
Emojis and the praise package
Okay, I have two questions that are a little bit of emoji pushback, not to be a killjoy, but praise seems unnecessary. Why not include it as a suggested package? And then also, is there a way to suppress emojis? Or build pipelines that may not support them?
Oh, so really, I don't know. So first of all, it doesn't use the praise package. It just uses its own. Well, I guess it does. Is that right? Let me refresh my memory. I guess it does use the praise package, which is a tiny package. So I guess we could make this a suggested package, but it just seems like, I don't know. If someone did a pull request that made praise into a suggested package rather than a dependent package, I'd consider it, but just doesn't, I don't think it practically really reduces the cost of installing, test that in any practical way.
If you don't want emojis, you can just turn, or you can't turn the emojis off, although they will be automatically turned off if you're not in a UTF-8 locale, which is normally on Windows. But you can test report. If I remember, what's it called? Progress reporter. If you really don't like them, you can turn them off by calling, setting stuff up in a special way and saying show praise equals false. But the praise only shows up in reporters that are used interactively. It never shows up in reporters that are used in our command check or in other continuous integration context. So I think it's like, to me, it's purely just a fun thing.
I don't think there's anything wrong with introducing a little fun and joy into unit testing and by and large, people really like it. But if you don't, you can turn it off.
Controlling waldo output
Okay. I've been using a waldo a bit. Is there any planned way to control the amount of output that is shown?
Yes. So someone actually just did a pull request for waldo so that you can choose to print any number of changes. By default, waldo doesn't show. It shows like 10 changes with the kind of assumption that if you have more than 10 changes, you probably want to start by fixing the first 10 changes and that will reveal more issues.
I think this is something I, this is just purely driven by pragmatics. You don't want failures that generate hundreds of lines of output. When there's something horrifically wrong in the first line that is actionable. So I don't have a good sense of where that slider should be set by default and how you should slide it to more or less output. Yet, that's something that we'll keep thinking about and iterating on as we work on this in reality in our packages.
Git branch in the R console prompt
All right. I've picked two last questions. One is super tiny and then one's a little bit bigger and makes a good conclusion. So the tiny off-topic question is, how do you get the git branch into your prompt in the R console?
Yeah. So if you're wondering what this thing was, this is the name of my active git branch and that is some code on my R profile that uses Gabor's prompt package to just put the git branch. And then this automatically updates whenever you run in the R code. So it automatically stays up to date as you change the branch.
Long-term plan for the edition system
Okay. So the last question, I think it's a good one to close on is to think about like, what's the long-term game plan with this addition idea? So this person's like, for now in the dev version, after I install it, I have to change this config entry in description. Will this still be the case indefinitely when it's not the dev version, but it's the released version or at some point, will it default to addition three and someone might have to opt in the version addition two?
I don't see any way to change the default edition from two without breaking hundreds of CRAN packages. So I think that implies that the default version will always be two. But what we will do in the tools that we use for creating packages, like and use this, we will automatically set the config to kind of the latest version. So when you start, the kind of the idea is that when you start a new package, you get the latest, you get the latest edition of test that when you are working on an old package, you have to deliberately opt into the new version if you really want it. I think that kind of offers the best compromise of packages that are actively developed can easily upgrade without touching anything that's that's not being developed.
I don't see any way to change the default edition from two without breaking hundreds of CRAN packages. So I think that implies that the default version will always be two.
Thanks for thanks for joining us and learning about the upcoming third edition of test that again, like now is the perfect time to give feedback on things you like, things that you hate. We're listening and we really appreciate appreciate your feedback.
