Unable to relay packets with error (serde parse error: missing field data)

[achilleas-kal]

Summary of Bug

Hermes v0.15 won't relay packets on Injective Chain (ibc-go v3.1) with the error below. Downgrading to hermes v0.13 fixed the issue.

Jul 14 18:21:10 i-fsn-01 hermes[1297]: 2022-07-14T16:21:10.116036Z WARN ThreadId(48) refresh{client=07-tendermint-29 src_chain=injective-1 dst_chain=columbus-5}: task encountered ignorable error: error raised while trying to refresh client 07-tendermint-29: failed querying the application status of source chain: RPC error to endpoint http://xxx.com:4093/: serde parse error: missing field data at line 8 column 5

Steps to Reproduce

This started happening upon a chain upgrade on Injective, the only change that could be related to IBC transfers was an upgrade from ibc-go v2 to ibc-go v3.1.

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[romac]

Looks like the data returned by the /abci_info RPC endpoint of Injective chains is missing a couple fields, namely data and version. This used to work with Hermes < v0.15 because we only started querying this endpoint in v0.15 to fix issue #1970.

As far as I can tell, we do not use the value of these two fields, so we could make them optional.

Injective

$ curl https://tm.injective.network/abci_info?

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "response": {
      "last_block_height": "13111281",
      "last_block_app_hash": "nf/g23bemW4IAgz53DqCliUivcra12DFw/I60URaIcA="
    }
  }
}

Cosmos Hub

$ curl https://rpc.cosmos.network/abci_info?

{
  "jsonrpc": "2.0",
  "id": -1,
  "result": {
    "response": {
      "data": "GaiaApp",
      "version": "v7.0.0",
      "last_block_height": "11338424",
      "last_block_app_hash": "FvCOSo05Doi2fuLkp66s79cm4a336Lti3pTylto/O6k="
    }
  }
}

[achilleas-kal]

@romac we looked into it and had to add this from MakeFile so should be fixed in production too on our end soon, it might be a good idea to make them optional though since it's a manual process that has to be done.

[romac]

Good to hear, thanks! We will indeed look into making those optional on our end.

[mzabaluev]

This looks similar to informalsystems/tendermint-rs#1132. I will backport informalsystems/tendermint-rs#1131 to tendermint-rs 0.23.

[adizere]

looked into it and had to add this from MakeFile so should be fixed in production too on our end soon, it might be a good idea to make them optional though since it's a manual process that has to be done.

Is it possible to provide further details here on what you changed to your Makefile, in case others encounter the same problem? Many thanks!

[achilleas-kal]

@adizere sure. These two lines did the trick

COSMOS_VERSION_NAME = injective

install: export VERSION_FLAGS="-X $(VERSION_PKG).AppVersion=$(APP_VERSION) -X $(VERSION_PKG).GitCommit=$(GIT_COMMIT)  -X $(VERSION_PKG).BuildDate=$(BUILD_DATE) -X $(COSMOS_VERSION_PKG).Version=$(APP_VERSION) -X $(COSMOS_VERSION_PKG).Name=$(COSMOS_VERSION_NAME) -X $(COSMOS_VERSION_PKG).AppName=injectived -X $(COSMOS_VERSION_PKG).Commit=$(GIT_COMMIT)"

[adizere]

@achilleas-kal would you consider quickly running Hermes from #2465 to see if this issue is still causing problems in production?

The upgrade to the newer tendermint-rs lib should have fixed it.
#2456

[achilleas-kal]

@adizere happy to but we're using kubernetes and we build docker images from the hermes repository so I can't exactly build from source. Could you push a docker image with the fix? If I get an image i.e. informalsystems/hermes:0.13.0 that includes the fix I can test quickly.

[mzabaluev]

@adizere sure. These two lines did the trick

COSMOS_VERSION_NAME = injective

install: export VERSION_FLAGS="-X $(VERSION_PKG).AppVersion=$(APP_VERSION) -X $(VERSION_PKG).GitCommit=$(GIT_COMMIT)  -X $(VERSION_PKG).BuildDate=$(BUILD_DATE) -X $(COSMOS_VERSION_PKG).Version=$(APP_VERSION) -X $(COSMOS_VERSION_PKG).Name=$(COSMOS_VERSION_NAME) -X $(COSMOS_VERSION_PKG).AppName=injectived -X $(COSMOS_VERSION_PKG).Commit=$(GIT_COMMIT)"

This does not change /abci_info output for the gaiad binary I'm trying to build locally (in a source tree with v7.0.2 checked out, which I also built earlier).

[romac]

Since @mzabaluev added test fixtures to tendermint-rs for the case where the version and/or data are missing, I think we can go ahead and close this issue. I'll include a changelog entry in the v1.0.0-rc.1 release PR.

[albertchon]

@mzabaluev Looks like @achilleas-kal didn't include the other relevant variables for the version flags (he only included our commit diff). Here's the portion for the relevant LDFLAGS from our Makefile

GIT_COMMIT = $(shell git rev-parse --short HEAD)
COSMOS_VERSION_PKG = github.com/cosmos/cosmos-sdk/version
COSMOS_VERSION_NAME = injective


# process build tags
build_tags = netgo
ifeq ($(LEDGER_ENABLED),true)
  ifeq ($(OS),Windows_NT)
    GCCEXE = $(shell where gcc.exe 2> NUL)
    ifeq ($(GCCEXE),)
      $(error gcc.exe not installed for ledger support, please install or set LEDGER_ENABLED=false)
    else
      build_tags += ledger
    endif
  else
    UNAME_S = $(shell uname -s)
    ifeq ($(UNAME_S),OpenBSD)
      $(warning OpenBSD detected, disabling ledger support (https://github.com/cosmos/cosmos-sdk/issues/1988))
    else
      GCC = $(shell command -v gcc 2> /dev/null)
      ifeq ($(GCC),)
        $(error gcc not installed for ledger support, please install or set LEDGER_ENABLED=false)
      else
        build_tags += ledger
      endif
    endif
  endif
endif

ifeq ($(WITH_CLEVELDB),yes)
  build_tags += gcc
endif
build_tags += $(BUILD_TAGS)
build_tags := $(strip $(build_tags))
whitespace :=
empty = $(whitespace) $(whitespace)
comma := ,
build_tags_comma_sep := $(subst $(empty),$(comma),$(build_tags))

install: export GOPROXY=direct
install: export VERSION_FLAGS="-X $(COSMOS_VERSION_PKG).Version=$(APP_VERSION) -X $(COSMOS_VERSION_PKG).Name=$(COSMOS_VERSION_NAME) -X $(COSMOS_VERSION_PKG).AppName=injectived -X $(COSMOS_VERSION_PKG).Commit=$(GIT_COMMIT)"
install:
	cd cmd/injectived/ && go install -tags $(build_tags_comma_sep) -ldflags $(VERSION_FLAGS)