Inconsistency between `testthat::test_file()` and `devtools::test()` with helpers and environment variables

[DanChaltiel]

Hi,

I'm using is_testing() in my helpers files to load some objects on load_all(), but not during tests.
This is very convenient in my field of work.

However, tests behave differently in RStudio when using the "Test" button in the Build pane, which seems to run devtools::test(), and the "Run tests" button in the source pane, which seems to run testthat::test_file():

• For devtools::test(), is_testing() is FALSE inside the helper
• For testthat::test_file(), is is TRUE

I'm posting this issue here because IMHO, testthat is right here (we are currently running tests).

I have up-to-date RStudio, devtools, and testthat (as per CRAN).

Minimal reproducible example

#in tests/testthat/test-poc.R  
test_that("devtools POC", {  
  expect_true(is_testing())  
})  

test_that("devtools POC2", {  
  expect_true(it)  
})  

#in tests/testthat/helper-poc.R  
it = is_testing()

Tests :

> testthat::test_file("tests/testthat/test-poc.R")
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 2 ]> 

> devtools::test()
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 1 ]

── Failure (test-poc.R:6:3): devtools POC2 ────────────────────────────────────────────────────────────────────────
`it` is not TRUE

`actual`:   FALSE
`expected`: TRUE 
[ FAIL 1 | WARN 0 | SKIP 0 | PASS 1 ]

Thank you for your work on this package!

PS: In a more complex project, unfortunately not reprexable, I had even discrepancies inside devtools, with all tests passing with devtools::test() but failing with devtools::test_active_file().

[jennybc]

I've made a small example package to poke at this:

https://github.com/jennybc/istesting

I think you're using is_testthat() in an unconventional way, by evaluating it at the top-level in a helper file and storing the result.

is_testthat() is documented like so:

is_testing() is TRUE inside a test.

And that is definitely true. I believe you're thinking of it as TRUE inside a test run, which is not universally true.

I'm finding that it does evaluate to TRUE in top-level code outside a test, in a test file. But I'd discourage having code in a test file that is outside of test_that() if you can possibly avoid it.

But it is_testing() does not evaluate to TRUE in top-level code in a helper file apparently. I see this:

> devtools::test()
ℹ Testing istesting
Run `rlang::last_trace()` to see where the error occurred.
Error in `FUN()`:
! In path: "/Users/jenny/tmp/istesting/tests/testthat/helper.R"
Caused by error:
! is_testing() is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Show Traceback

So my recommendation is to adjust your usage of is_testing() to be better aligned with the use case it's designed for, i.e. make sure it's going to be executed when the test body executes.

[DanChaltiel]

Thank you very much for your answer @jennybc!

The issue was mostly about the inconsistency between testthat and devtools, which can be confusing from a user perspective.

Personally, I think it's quite reasonable to assume that is_testing() returns TRUE during a test run — at least at the top level of files in the tests/ directory.

Since you kindly addressed my usage, I wanted to share a bit more detail.
This feature is so convenient that I really hope I’m not the only one relying on it. I’ll admit, I’m a bit terrified that my issue might lead to changing this behavior!

I work on confidential data, and I build packages to handle it.
In a git-ignored helper file, I write:

if(!is_testing() && !is_checking()){
  data = read_confidential_data("gitignored_path/")
  function_im_working_on(data) |> print()
}

This setup means that every time I make a change and reload my environment, the function I’m currently working on runs immediately, showing the updated result or hitting a browser() call if needed.

Re-running the whole test suite each time would require a much better programming environment (and stronger discipline!) than what I currently have, and it would slow down day-to-day work considerably.

Thanks again for taking the time and for all your work on these tools!

[jennybc]

I did actually get further in a diagnosis, but it does not change my advice:

adjust your usage of is_testing() to ... make sure it's going to be executed when the test body executes

But since I know the reason, I want to record it now for posterity! It's not a difference between testthat and devtools. It's a difference between testthat::test_local() and other testthat ways to run some or all of your tests.

In my demo package, I noticed that a simple load_all() errors! Because it sources the test helper and this top-level code in helper.R throws: expect_true(is_testing()).

https://github.com/jennybc/istesting/blob/2cb187ea26fde82f4ba8f4487683d775203861e9/tests/testthat/helper.R#L5C1-L5C26

devtools::test() is basically just a convenience wrapper for testthat::test_local():

https://github.com/r-lib/testthat/blob/1615d4ccf5e8d8a487028396fae07f8e7f5369f0/R/test-package.R#L62-L74

which kicks off with package <- pkgload::pkg_name(path). The test helper code is run first, then test_dir() is called. So that explains why storing the result of is_testing() in a helper file is suboptimal.

(I doubt the conclusion for you is that you always have to run the whole test suite. I suspect there's just a different way to rig things, so that is_testing() is being run at the "right" time.)