Fix nondeterministic test failures

[pokey]

Let's start by just retrying tests 3 times if they fail. This approach is terrible, because it swallows all nondeterministic bugs, but at least we can start requiring CI passes, unlike now where they're disabled because they fail

Originally posted by @pokey in #172 (comment)

[pokey]

@AndreasArvidsson @brxck lets move the sleep discussion here. Here's @AndreasArvidsson's comment from the PR:

@brxck Is it only mac that needs the sleep time? If that's the case we could probably do something like this. Not as optimal as removing the sleep totally foriall platforms but still better than having them where they are not needed.

if (os.platform() === "darwin") {

        await new Promise((resolve) => setTimeout(resolve, 100));

 }

[pokey]

@AndreasArvidsson It's actually only macOS in CI, so we might be able to be even more specific by checking env var to see if CI as well as your os check

It's quite odd it needs a sleep at all tbh because there's nothing async going on there. It's just setting some stuff in an object.

[AndreasArvidsson]

Then that sounds like a good solution :)

I've already pushed the operating system change. Which env var would be the easiest way to detect CI?

[pokey]

CI. See https://docs.github.com/en/actions/reference/environment-variables

[brxck]

I'll say that tests on Ubuntu occasionally had the same problem, including my own (not slow) machine.

It is a weird problem... I had an alternative idea of adding retry logic to where we check the navigationMap?

[AndreasArvidsson]

@pokey I should probably work

if (process.platform === "darwin" && process.env["CI"]) {
        await new Promise((resolve) => setTimeout(resolve, 100));
}

@brxck Okay then above might not be a good solution after all.

[pokey]

Yeah given @brxck 's comment I think exponential backoff is the way to go

[AndreasArvidsson]

A retry logic sounds like a good idea.

[AndreasArvidsson]

A retry logic is now implemented. Doing three tries with 100 mill seconds between each.

[pokey]

Looks like that still failed; I switched it to exponential so hopefully that's more reliable

#174

[pokey]

Weird that didn't seem to help. Maybe it's not a timing thing? Maybe it's actually just nondeterministic for some reason? We should figure out how get more useful output from the assertion

[AndreasArvidsson]

Yes if a three second sleep doesn't help it's not that.

[pokey]

Yeah the first question to figure out is whether it is not doing the decorations or if it is doing them but they don't match our expectations

[pokey]

I kind of have the feeling that the sleep makes no difference 😂

[AndreasArvidsson]

Well that's just great then :D