Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Flaky Tests in TestSync_Basic #2165

Closed
Tracked by #1975
danforbes opened this issue Jan 10, 2022 · 1 comment · Fixed by #2470
Closed
Tracked by #1975

Fix Flaky Tests in TestSync_Basic #2165

danforbes opened this issue Jan 10, 2022 · 1 comment · Fixed by #2470
Assignees

Comments

@danforbes
Copy link
Contributor

danforbes commented Jan 10, 2022

Logs

@danforbes danforbes mentioned this issue Jan 10, 2022
13 tasks
@danforbes danforbes changed the title TestSync_Basic (logs) Fix Flaky Tests in TestSync_Basic Jan 10, 2022
@qdm12
Copy link
Contributor

qdm12 commented Apr 4, 2022

This was done indirectly (branch qdm12/tests-refactor-17 not in PR review yet, waiting for previous branches to be merged)

@qdm12 qdm12 self-assigned this Apr 4, 2022
qdm12 added a commit that referenced this issue Jun 10, 2022
- make RPC helper functions not test aware
- `waitForNode` context based function
- Use `t.TempDir()` for nodes base paths
- Add `GenerateGenesisAuths` helper function
- Remove global variables
  - `GenesisDefault`, `GenesisDev`, `GenesisTwoAuthsSecondaryVRF0_9_10`
  - framework variable
  - configuration variables
  - `HOSTNAME` variable
  - `maxTries` in stress tests
  - `testTimeout` in stress tests
- Write node configurations to `t.TempDir()` per test
- Remove `DecodeRPC_NT` function
- Split `tests/utils` package
  - `tests/utils/rpc` package
  - `tests/utils/websocket` package
  - `tests/utils/config` package
- Rework start and stop of nodes
  - Init nodes in parallel
  - Remove unneeded logs
  - Log when node is ready
  - Prefix writer with node string for multiple nodes
  - Wait for nodes after starting all nodes
- Add test writer using `t.Logf`
- Refactor `getResponse`
  - not test aware
  - get passed target interface
  - push test assertion/skip to actual test
- Add helper `retry.UntilNoError` and `retry.UntilOK` functions
  - Use in `compareBlocksByNumber`
  - Use in `waitForNode`
  - Change `GetBlockHash` to not retry RPC
  - Remove `PostWithRetry`
  - Replace `compareFinalizedHeadsWithRetry` with `retry.UntilOK`
- Refactor and fix `TestSystemRPC`
  - Add empty skipped tests for missing cases
  - Split each subtest individually
  - Keep on retrying until main context is canceled
  - Fix `networkState` test case
  - Assert more fields
  - Fix #2161 and #807
- Refactor and fix `TestSync_SingleBlockProducer`
- Refactor and make faster `TestAuthorSubmitExtrinsic`
- Configure nodes using TOML Go struct only
- Pass `--no-telemetry` as flag since it's not in TOML config
- `dot`: remove `exportConfig` and `TestExportConfig`
- `dot`: remove `ExportTomlConfig` and its test
- Install NPM dependencies from Go code for polkdaot JS test
- Remove `websocket` package
  - Use `Decode` from rpc package instead
  - Add fields to RPC server response struct
- Do not use `reflect`
  - Add `fetchWithTimeout` helper function
  - Change test cases slices to individual subtests
  - Add `t.SkipNow()` to parent tests where all subtests are skipped
  - Make `TestChainRPC` event driven and increase testing depth
  - Add a few `TODO`s to increase testing depth
  - Run each query call in parallel
- Refactor `TestChainSubscriptionRPC`
  - Refactor websocket code
  - Assert things and increase test depth
  - Graceful shutdown of websocket
- Merge stable workflow in rpc workflow
  - Use Go to implement `integration-test-all.sh` directly in single 'stable' test
  - The test is fast and only does RPC calls, it should be part of the RPC workflow
  - Remove stable workflow from Makefile, github workflows directory and document
- Log out command, logs and configuration file content on runtime error
- Fixes directly related issues #2389, #2390, #2391
- Fixes flaky tests issues #807, #2161, #2165, #2167

Co-authored-by: Timothy Wu <timwu20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants