Indrajeet Patil
The goal of a unit test is to capture the expected output of a function using code and making sure that actual output after any changes matches the expected output.
{testthat}
is a popular framework for writing unit tests in R.
Benefits of unit testing
Test output
Test pass only when actual function behaviour matches expected.
actual | expected | tests |
---|---|---|
{testthat}
: A recapTest organization
Testing infrastructure for R package has the following hierarchy:
Component | Role |
---|---|
Test file |
Tests for R/foo.R will typically be in tests/testthat/test-foo.R . |
Tests | A single file can contain multiple tests. |
Expectations |
A single test can have multiple expectations. |
Example test file
Every test is a call to testthat::test_that()
function.
Every expectation is represented by testthat::expect_*()
function.
You can generate a test file using usethis::use_test()
function.
A unit test records the code to describe expected output.
(actual) (expected)
A snapshot test records expected output in a separate, human-readable file.
(actual) (expected)
If you develop R packages and have struggled to
then you should be excited to know more about snapshot tests (aka golden tests)! 🤩
Familiarity with writing unit tests using {testthat}
.
If not, have a look at this chapter from R Packages book.
Snapshot tests can be used to test that text output prints as expected.
Important for testing functions that pretty-print R objects to the console, create elegant and informative exceptions, etc.
Let’s say you want to write a unit test for the following function:
Source code
Note that you want to test that the printed output looks as expected.
Therefore, you need to check for all the little bells and whistles in the printed output.
Even testing this simple function is a bit painful because you need to keep track of every escape character, every space, etc.
test_that("`print_movies()` prints as expected", {
expect_equal(
print_movies(
c("Title", "Director"),
c("Salaam Bombay!", "Mira Nair")
),
"Movie: \n Title: Salaam Bombay!\n Director: Mira Nair"
)
})
Test passed 🥳
With a more complex code, it’d be impossible for a human to reason about what the output is supposed to look like.
Important
If this is a utility function used by many other functions, changing its behaviour would entail manually changing expected outputs for many tests.
This is not maintainable! 😩
Instead, you can use expect_snapshot()
, which, when run for the first time, generates a Markdown file with expected/reference output.
test_that("`print_movies()` prints as expected", {
local_edition(3)
expect_snapshot(cat(print_movies(
c("Title", "Director"),
c("Salaam Bombay!", "Mira Nair")
)))
})
── Warning: `print_movies()` prints as expected ────────────────────────────────
Adding new snapshot:
Code
cat(print_movies(c("Title", "Director"), c("Salaam Bombay!", "Mira Nair")))
Output
Movie:
Title: Salaam Bombay!
Director: Mira Nair
Warning
The first time a snapshot is created, it becomes the truth against which future function behaviour will be compared.
Thus, it is crucial that you carefully check that the output is indeed as expected. 🔎
Compared to your unit test code representing the expected output
notice how much more human-friendly the Markdown output is!
Code
cat(print_movies(c("Title", "Director"), c("Salaam Bombay!", "Mira Nair")))
Output
Movie:
Title: Salaam Bombay!
Director: Mira Nair
It is easy to see what the printed text output is supposed to look like. In other words, snapshot tests are useful when the intent of the code can only be verified by a human.
More about snapshot Markdown files
If test file is called test-foo.R
, the snapshot will be saved to test/testthat/_snaps/foo.md
.
If there are multiple snapshot tests in a single file, corresponding snapshots will also share the same .md
file.
By default, expect_snapshot()
will capture the code, the object values, and any side-effects.
If you run the test again, it’ll succeed:
test_that("`print_movies()` prints as expected", {
local_edition(3)
expect_snapshot(cat(print_movies(
c("Title", "Director"),
c("Salaam Bombay!", "Mira Nair")
)))
})
Test passed 🌈
Why does my test fail on a re-run?
If testing a snapshot you just generated fails on re-running the test, this is most likely because your test is not deterministic. For example, if your function deals with random number generation.
In such cases, setting a seed (e.g. set.seed(42)
) should help.
When function changes, snapshot doesn’t match the reference, and the test fails:
Changes to function
print_movies <- function(keys, values) {
paste0(
"Movie: \n",
paste0(
" ", keys, "- ", values,
collapse = "\n"
)
)
}
Failure message provides expected (-
) vs observed (+
) diff.
Test failure
test_that("`print_movies()` prints as expected", {
expect_snapshot(cat(print_movies(
c("Title", "Director"),
c("Salaam Bombay!", "Mira Nair")
)))
})
── Failure: `print_movies()` prints as expected ────────────────────────────────
Snapshot of code has changed:
old[2:6] vs new[2:6]
cat(print_movies(c("Title", "Director"), c("Salaam Bombay!", "Mira Nair")))
Output
Movie:
- Title: Salaam Bombay!
+ Title- Salaam Bombay!
- Director: Mira Nair
+ Director- Mira Nair
* Run `testthat::snapshot_accept('slides.qmd')` to accept the change.
* Run `testthat::snapshot_review('slides.qmd')` to interactively review the change.
Error:
! Test failed
Message accompanying failed tests make it explicit how to fix them.
Fixing multiple snapshot tests
If this is a utility function used by many other functions, changing its behaviour would lead to failure of many tests.
You can update all new snapshots with snapshot_accept()
. And, of course, check the diffs to make sure that the changes are expected.
So far you have tested text output printed to the console, but you can also use snapshots to capture messages, warnings, and errors.
message
Tip
Snapshot records both the condition and the corresponding message.
You can now rest assured that the users are getting informed the way you want! 😌
In case of an error, the function expect_snapshot()
itself will produce an error. You have two ways around this:
Option-1 (recommended)
── Warning: `log()` errors ─────────────────────────────────────────────────────
Adding new snapshot:
Code
log("x")
Condition
Error in `log()`:
! non-numeric argument to mathematical function
Which option should I use?
If you want to capture both the code and the error message, use expect_snapshot(..., error = TRUE)
.
If you want to capture only the error message, use expect_snapshot_error()
.
testthat article on snapshot testing
Introduction to golden testing
Docs for Jest library in JavaScript, which inspired snapshot testing implementation in testthat
To create graphical expectations, you will use testthat extension package: {vdiffr}
.
{vdiffr}
work?vdiffr introduces expect_doppelganger()
to generate testthat expectations for graphics. It does this by writing SVG snapshot files for outputs!
The figure to test can be:
ggplot
object (from ggplot2::ggplot()
)recordedplot
object (from grDevices::recordPlot()
)print()
methodNote
If test file is called test-foo.R
, the snapshot will be saved to test/testthat/_snaps/foo
folder.
In this folder, there will be one .svg
file for every test in test-foo.R
.
The name for the .svg
file will be sanitized version of title
argument to expect_doppelganger()
.
Let’s say you want to write a unit test for the following function:
Source code
Note that you want to test that the graphical output looks as expected, and this expectation is difficult to capture with a unit test.
You can use expect_doppelganger()
from vdiffr to test this!
The first time you run the test, it’d generate an .svg
file with expected output.
Warning
The first time a snapshot is created, it becomes the truth against which future function behaviour will be compared.
Thus, it is crucial that you carefully check that the output is indeed as expected. 🔎
You can open .svg
snapshot files in a web browser for closer inspection.
If you run the test again, it’ll succeed:
When function changes, snapshot doesn’t match the reference, and the test fails:
Changes to function
Test failure
test_that("`create_scatter()` plots as expected", {
local_edition(3)
expect_doppelganger(
title = "create scatter",
fig = create_scatter(),
)
})
── Failure ('<text>:3'): `create_scatter()` plots as expected ──────────────────
Snapshot of `testcase` to 'slides.qmd/create-scatter.svg' has changed
Run `testthat::snapshot_review('slides.qmd/')` to review changes
Backtrace:
1. vdiffr::expect_doppelganger(...)
3. testthat::expect_snapshot_file(...)
Error in `reporter$stop_if_needed()`:
! Test failed
Running snapshot_review()
launches a Shiny app which can be used to either accept or reject the new output(s).
Why are my snapshots for plots failing?! 😔
If tests fail even if the function didn’t change, it can be due to any of the following reasons:
For these reasons, snapshot tests for plots tend to be fragile and are not run on CRAN machines by default.
Whole file snapshot testing makes sure that media, data frames, text files, etc. are as expected.
Let’s say you want to test JSON files generated by jsonlite::write_json()
.
Test
# File: tests/testthat/test-write-json.R
test_that("json writer works", {
local_edition(3)
r_to_json <- function(x) {
path <- tempfile(fileext = ".json")
jsonlite::write_json(x, path)
path
}
x <- list(1, list("x" = "a"))
expect_snapshot_file(r_to_json(x), "demo.json")
})
── Warning: json writer works ──────────────────────────────────────────────────
Adding new file snapshot: 'tests/testthat/_snaps/demo.json'
Snapshot
Note
To snapshot a file, you need to write a helper function that provides its path.
If a test file is called test-foo.R
, the snapshot will be saved to test/testthat/_snaps/foo
folder.
In this folder, there will be one file (e.g. .json
) for every expect_snapshot_file()
expectation in test-foo.R
.
The name for snapshot file is taken from name
argument to expect_snapshot_file()
.
If you run the test again, it’ll succeed:
If the new output doesn’t match the expected one, the test will fail:
# File: tests/testthat/test-write-json.R
test_that("json writer works", {
local_edition(3)
r_to_json <- function(x) {
path <- tempfile(fileext = ".json")
jsonlite::write_json(x, path)
path
}
x <- list(1, list("x" = "b"))
expect_snapshot_file(r_to_json(x), "demo.json")
})
── Failure: json writer works ──────────────────────────────────────────────────
Snapshot of `r_to_json(x)` to 'slides.qmd/demo.json' has changed
Run `testthat::snapshot_review('slides.qmd/')` to review changes
Error:
! Test failed
Running snapshot_review()
launches a Shiny app which can be used to either accept or reject the new output(s).
Documentation for expect_snapshot_file()
To write formal tests for Shiny applications, you will use testthat extension package: {shinytest2}
.
{shinytest2}
work?shinytest2 uses a Shiny app (how meta! 😅) to record user interactions with the app and generate snapshots of the application’s state. Future behaviour of the app will be compared against these snapshots to check for any changes.
Exactly how tests for Shiny apps in R package are written depends on how the app is stored. There are two possibilities, and you will discuss them both separately.
Stored in /inst
folder
├── DESCRIPTION
├── R
├── inst
│ └── sample_app
│ └── app.R
Returned by a function
├── DESCRIPTION
├── R
│ └── app-function.R
├── DESCRIPTION
├── R
├── inst
│ └── sample_app
│ └── app.R
Let’s say this app resides in the inst/unitConverter/app.R
file.
To create a snapshot test, go to the app directory and run record_test()
.
Test
Snapshot
Note
record_test()
will auto-generate a test file in the app directory. The test script will be saved in a subdirectory of the app (inst/my-app/tests/testthat/test-shinytest2.R
).
There will be one /tests
folder inside every app folder.
The snapshots are saved as .png
file in tests/testthat/test-shinytest2/_snaps/{.variant}/shinytest2
. The {.variant}
here corresponds to operating system and R version used to record tests. For example, _snaps/windows-4.1/shinytest2
.
Note that currently your test scripts and results are in the /inst
folder, but you’d also want to run these tests automatically using testthat.
For this, you will need to write a driver script like the following:
library(shinytest2)
test_that("`unitConverter` app works", {
appdir <- system.file(package = "package_name", "unitConverter")
test_app(appdir)
})
Now the Shiny apps will be tested with the rest of the source code in the package! 🎊
Tip
You save the driver test in the /tests
folder (tests/testthat/test-inst-apps.R
), alongside other tests.
Let’s say, while updating the app, you make a mistake, which leads to a failed test.
Changed code with mistake
ui <- fluidPage(
titlePanel("Convert kilograms to grams"),
numericInput("kg", "Weight (in kg)", value = 0),
textOutput("g")
)
server <- function(input, output, session) {
output$g <- renderText(
paste0("Weight (in kg): ", input$kg * 1000) # should be `"Weight (in g): "`
)
}
shinyApp(ui, server)
Test failure JSON diff
Fixing this test will be similar to fixing any other snapshot test you’ve seen thus far.
{testthat2}
provides a Shiny app for comparing the old and new snapshots.
├── DESCRIPTION
├── R
│ └── app-function.R
The only difference in testing workflow when Shiny app objects are created by functions is that you will write the test ourselves, instead of shinytest2 auto-generating it.
Source code
# File: R/unit-converter.R
unitConverter <- function() {
ui <- fluidPage(
titlePanel("Convert kilograms to grams"),
numericInput("kg", "Weight (in kg)", value = 0),
textOutput("g")
)
server <- function(input, output, session) {
output$g <- renderText(
paste0("Weight (in g): ", input$kg * 1000)
)
}
shinyApp(ui, server)
}
you call record_test()
directly on a Shiny app object, copy-paste commands to the test script, and run devtools::test_active_file()
to generate snapshots.