end-to-end tests: refactor and fix flaky tests

[qdm12]

💡 Review commit by commit by starting with the first commit

Changes

TLDR

• Fix flakky tests and add testing depth
• More robust code
• Detect runtime errors
• Better concerns separation
• Can easily run a single test independently & locally (after make gossamer)

Details

• Faster TestAuthorSubmitExtrinsic
• Fix TestSync_SingleBlockProducer (Fix Flaky Tests in TestSync_SingleBlockProducer #2167)
• refactor compareBlocksByNumber
• refactor TestSystemRPC (Fix Flaky Tests in TestSystemRPC #2161)
  • add empty skipped tests for missing cases
  • split each subtest individually
  • keep on retrying until main context is canceled
  • fix networkState test case
  • assert more fields
  • fix Fix Flaky Tests in TestSystemRPC #2161 and RPC tests for system_health should start 2 peers and assert correctly #807
• retry.UntilNoError function
  • Use in compareBlocksByNumber
  • Use in waitForNode
  • change GetBlockHash to not retry RPC
  • remove PostWithRetry
• remove global variables maxTries and testTimeout in stress tests
• refactor getResponse
  • not test aware (tests/utils: make helping functions not test aware #2389)
  • get passed target interface
  • push test assertion/skip to actual test
• rework start and stop of nodes (tests/utils: refactor nodes starting and stopping and detect runtime errors #2391)
  • Remove unneeded logs
  • Prefix writer with node string for multiple nodes
  • Test writer using t.Logf
  • Wait for nodes after starting all nodes
  • Log when node is ready only
  • Init nodes in parallel
• split tests/utils package
  • tests/utils/rpc package
  • tests/utils/websocket package
  • tests/utils/config package
• remove DecodeRPC_NT
• remove HOSTNAME variable
• inline RPC method strings
• remove global variable GenesisTwoAuthsSecondaryVRF0_9_10
• remove global configuration variables
  • write no auth config to temp dir per test
  • write no grandpa config to temp dir per test
  • write no babe config to temp dir per test
  • write grandpa config to temp dir per test
  • write default config to temp dir per test
• remove global framework variable
• remove global variables GenesisDefault and GenesisDev
• Add GenerateGenesisAuths function
• use t.TempDir() for base paths
• waitForNode context based function (tests/utils: add waitForNode context-based function to wait for a node #2390)
• make RPC helpers not test aware (tests/utils: make helping functions not test aware #2389)
• make DecodeRPC not test aware (tests/utils: make helping functions not test aware #2389)

Tests

• RPC suite passing
• Stress tests passing

Issues

See issues mentioned in changes above.

Indirectly fixed: #2165

Primary Reviewer

[codecov]

Codecov Report

Merging #2470 (c1271b2) into development (6d22c23) will increase coverage by 0.00%.
The diff coverage is 28.07%.

@@             Coverage Diff              @@
##           development    #2470   +/-   ##
============================================
  Coverage        57.34%   57.34%           
============================================
  Files              215      215           
  Lines            28417    28407   -10     
============================================
- Hits             16296    16291    -5     
+ Misses           10442    10440    -2     
+ Partials          1679     1676    -3     

Flag	Coverage Δ
unit-tests	57.34% <28.07%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dot/utils.go	50.00% <ø> (+16.66%)	⬆️
lib/runtime/constants.go	0.00% <ø> (ø)
lib/runtime/test_helpers.go	7.94% <0.00%> (-0.65%)	⬇️
lib/utils/utils.go	17.48% <0.00%> (-0.20%)	⬇️
dot/peerset/peerset.go	49.34% <42.85%> (-0.23%)	⬇️
dot/state/epoch.go	49.53% <50.00%> (+0.23%)	⬆️
lib/runtime/life/test_helpers.go	100.00% <100.00%> (ø)
lib/runtime/wasmer/test_helpers.go	100.00% <100.00%> (ø)
dot/network/host.go	64.31% <0.00%> (-1.07%)	⬇️
lib/trie/database.go	50.30% <0.00%> (-0.93%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2fa5d8a...c1271b2. Read the comment docs.