chore: add back some --reruns to avoid flaky canary tests

[dkphm]

Which issue(s) does this change fix?

N/A

Why is this change necessary?

To fix flaky canary tests

How does it address the issue?

What side effects does this change have?

Mandatory Checklist

PRs will only be reviewed after checklist is complete

• Add input/output type hints to new functions/methods
• Write design document if needed (Do I need to write a design document?)
• Write/update unit tests
• Write/update integration tests
• Write/update functional tests if needed
• make pr passes
• make update-reproducible-reqs if dependencies were changed
• Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.