Run a tendermint node in CI to test the RPC & light client

[ebuchman]

As per some RPC incompatibilities being discovered, namely #88 and #4, we should have the CI spin up a tendermint node, eg. using the latest docker image, and run the currently ignored integration tests against it (via cargo test -- --ignored) - see #88 (comment)

[liamsi]

Likewise for the light node.

Basically these 2 things need a tendermint instance to run / test against:

1. Light node:

• Current light client main binary:
  https://github.com/interchainio/tendermint-rs/blob/master/tendermint-lite/src/main.rs
• And after refactoring (after Abscissify light node #125 gets merged):
  https://github.com/interchainio/tendermint-rs/blob/9c4fd2fb953aab71d4073c4fac106027073b3b5b/light-node/src/bin/light_node/main.rs
  I can probably use the acceptance tests against a tendermint instance instead of calling the main in a script: https://github.com/interchainio/tendermint-rs/blob/9c4fd2fb953aab71d4073c4fac106027073b3b5b/light-node/tests/acceptance.rs

2. Currently ignored integration tests:
   https://github.com/interchainio/tendermint-rs/blob/master/tendermint/tests/integration.rs

Ref: #144, informalsystems/hermes#22, #95

Maybe make it optional for the (ignored) tests to pass initially. So we can see the error output and fix the rpc issues and the (unknown) light node bugs.

[greg-szabo]

I also opened an issue in the tendermint-go repository to update the tendermint docker image so we can use it in GitHub Actions. This would make the conifguration simpler and we could get away from the custom tendermint docker image.

[greg-szabo]

Opened PR to update the tendermint-go Dockerfile.

[greg-szabo]

The new Dockerfile was merged to master in the tendermint repository. This means that changes will be available in the next 0.33 release.
I propose we keep the current custom docker image for 0.32 and by the time we upgrade our code to 0.33, hopefully there will be a new 0.33 release with the changes - and we can move over.

[greg-szabo]

I've been trying to run the light_node acceptance tests but I get bisect errors just starting the light_node. I'm going to park working on this while the code (or me) gets fixed.
Running a full node on the dev machine is slightly easier now because you can run a full node on your machine with one command, exactly how the CI does it:
docker run -it --rm -p 26657:26657 interchainio/tendermint. (Note: this will change when we upgrade to tendermint 0.33)
I'll reach out to @liamsi for further steps.

[greg-szabo]

Light client CI blocks on #186 . @liamsi is aware, we'll troubleshoot further as time permits.

[zmanian]

This to me looks a lot like the implicit clock sync issues we found in the IBC relayer light client? You are starting a new chain and new light client in different VMs and their clocks will likely be slightly out of sync. Introducing a delay of 5 -10 seconds before starting the light client helped in relayer development.

[liamsi]

I think what Greg found is a much simpler and more substantial bug (but I haven't verify it yet). But we might run into the problem you described (when this bug is fixed).

[ebuchman]

We should just need the genesis time for the chain we're using to be recent enough. We can also make the default trusting period longer. In any case doesn't seem like there's a bug, just a constraint on how we set up the test environment.

Note if we wanted the test to work against an existing public chain (eg hub-3), the client wouldn't be able to start from the first block. We'd have to provide a more recent height to initialize from (this couldn't be static, we'd want to do something like check a full node for a recent height and validator hash and use that to initialize the light client ...)

[liamsi]

I have a suggestion on how to further improve the integration test situation:

Currently, we always run against the lates (go) tendermint docker image. On one hand this is really cool because we get early warnings for things that will break compatibility with the next release. On the other hand, we can't guarantee compatibility against a current release (e.g. we only want compatibility with 0.33.X or even only with 0.33.4 because that's what the hub is using or goz or whatever reasons there might be to want compatibility to a particular version).

I think tests would be even more useful if:

1. they were run against a fixed and chosen tendermint version and were made mandatory (required on gh) for this version
2. additionally integration tests for the latest image run regularly too (but much less frequently; e.g. nightly/weekly)

Regarding 1): we currently do not merge PRs unless integration tests pass anyways. The tools could better reflect this already established process and only allow merging if intgration tests pass too. Also, sometimes integration tests pass on the branch but fail on master (#249 #233 and many more). This happens because there was a new docker image released between last check on branch and merge to master. This would be mitigated if we made sure, we commit to compatibility of one tendermint version.

And 2) would help us to be learn about upcoming changes on a regular basis.

[liamsi]

related:
#304 (comment)

[brapse]

This was resolved by closing #304

[liamsi]

Did I miss that we now spin up a light node (or an simple light client) against a (go) tendermint node? If not, I would reopen this issue. Or maybe it is better to open a new more focused one and link in your meta test-issue #320.

[ebuchman]

As far as I can tell we do not do that. Opened #373