Async test being run in parallel despite #[serial]

[torfsen]

I have a test case that is marked #[serial] and still runs in parallel with another test case.

The code:

fn main() {
    println!("Hello, world!");
}

#[cfg(test)]
mod tests {
    use std::time::Duration;
    use once_cell::sync::Lazy;  // 1.4.0
    use serial_test::serial;  // 0.5.1
    use tokio::sync::Mutex;  // 1.13.0

    static MUTEX: Lazy<Mutex<()>> = Lazy::new(Mutex::default);

    #[tokio::test]
    async fn test_a() {
        println!("A START");
        let lock = MUTEX.try_lock();
        assert!(lock.is_ok());
        tokio::time::sleep(Duration::from_secs(1)).await;
        println!("A END");
    }

    #[tokio::test]
    #[serial]
    async fn test_b() {
        println!("B START");
        let lock = MUTEX.try_lock();
        assert!(lock.is_ok());
        tokio::time::sleep(Duration::from_secs(1)).await;
        println!("B END");
    }
}

Both test cases try to acquire the same mutex, but since one of them is marked #[serial] I would expect that to work (my real use case is different but hard to boil down to such a simple example). However, here's the output of a test run:

$ cargo test -- --nocapture
    Finished test [unoptimized + debuginfo] target(s) in 0.03s
     Running unittests (target/debug/deps/serial_test_debug-3633b304ba0aaa5e)

running 2 tests
A START
B START
thread 'tests::test_b' panicked at 'assertion failed: lock.is_ok()', src/main.rs:28:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tests::test_b ... FAILED
A END
test tests::test_a ... ok

failures:

failures:
    tests::test_b

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.00s

error: test failed, to rerun pass '--bin serial_test_debug'

From the output we can clearly see that the test cases were running in parallel, and hence one of them failed to obtain the lock.

Is this a bug or a misunderstanding on my side?

[palfrey]

Is this a bug or a misunderstanding on my side?

Misunderstanding. The docs say "Multiple tests with the serial attribute are guaranteed to be executed in serial", which happened here. However, the tests without the serial attribute have no such guarantees!

If you've got any suggestions for improved wording though, that can be fixed.

[torfsen]

@palfrey: Thanks for the clarification!

I guess the actual wording is not that important, but I do think it should be mentioned. Here's what I would write (addition in bold):

Multiple tests with the serial attribute are guaranteed to be executed in serial. Ordering of the tests is not guaranteed however. Tests without the serial attribute may run at any time, including in parallel to tests marked serial.

Regarding my original use case: I have a single test case which needs to be run on its own, i.e. no other test case should run in parallel to it. All other test cases, however, may freely run in parallel with each other.

If I understand you correctly then this is not directly achievable using serial_test, right? My only choice would be to mark all tests as serial, which would guarantee that my special case runs on its own but would also lose me any parallelism for the other cases.

[torfsen]

In case anyone else stumbles upon this issue: I've solved it using a global tokio::sync::RwLock. The test case which must run on its own acquires a write lock, whereas all other tests acquire read locks.

@palfrey: I think it would be a nice addition to serial_test to have a #[parallel] marker which can be used like a read-lock in that approach: tests marked with #[serial] would never run at the same time as another test marked with either #[serial] or #[parallel] (i.e. #[serial] is equivalent to a write-lock). Multiple tests marked with #[parallel] could run at the same time as long as no #[serial] test is running. Unmarked test with neither #[serial] nor #[parallel] would run at any time. Being a Rust beginner I have no idea how hard that would be to implement within serial_test, though.

[palfrey]

#[parallel] looks like an interesting option. Internally we use parking_lot's ReentrantMutex, and the RwLock doesn't allow for recursive write so it's not quite the same, but can probably be done with a bit more thinking about it.

Something to look at in the future, certainly. I'm currently looking at file locking but will see if I can find some time for this after that.

[torfsen]

@palfrey Great. I'll leave this ticket open in case you want to keep it as a feature request for #[parallel]. However, since I found a workaround that works for me, feel free to close it.

[nnethercote]

I had the same question and issue about mixing serial and non-serial tests. (Context is here if you want to know more; you'll need to read both forwards and backwards from that comment to fully understand.)

Using a RwLock is a clever idea, I might end up using that for my own purposes.

The parallel annotation would be nice, but I can see it would be a decent amount of work. In the meantime, adding @torfsen's documentation suggestion "Tests without the serial attribute may run at any time, including in parallel to tests marked serial" would be helpful.

[palfrey]

FYI @torfsen @nnethercote - there is now a PR #54 that implements #[parallel]. I think it works as described here (although I haven't documented it yet) and it's had some testing, but if either of you have any additional things you can test it against at some point, that'd be most appreciated!

It currently breaks on Windows builds, but not for any good reason I can yet figure out

[palfrey]

I've figured out the Windows issue (well mostly), in that apparently it's Condvar's don't always signal properly, and adding a timeout to their waits sorted things. I've also documented #[parallel].

AFAIK this works and does all the correct things. I'm going to merge #54 and release 0.7.0 with that in about a week's time if nothing else has been found, but if anyone listening here feels like running that branch against their use cases, it'd be appreciated!