Make go.thethings.network/lorawan-stack/v3/cmd a module

[johanstokking]

Summary

Replaces #2493

Make go.thethings.network/lorawan-stack/v3/cmd a module

Changes

• Move tools.go into new tools mod
• Make cmd a module
• Build development tooling to .bin instead of go run which would use the repo's module

Notes for Reviewers

1. Check need for the replacements in /go.mod and /cmd/go.mod
2. Make sure that we can use the tools in /cmd/tools.go. I'm also open to a /tools module
3. Check if we need to merge some changes from github.com/robertkrimen/otto in github.com/TheThingsIndustries/otto
4. Figure out how we can update .bin binaries if modules update in tools/go.mod; maybe always install on make init?

Checklist

• Scope: The referenced issue is addressed, there are no unrelated changes.
• Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md. The target branch is set to master if the changes are fully compatible with existing API, database, configuration and CLI.
• Testing: The changes are covered with unit tests. The changes are tested manually as well.
• Documentation: Relevant documentation is added or updated.
• Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
• Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

[johanstokking]

Blocked by #2496

[johanstokking]

In order to use versioned tools, I would suggest;

• put tools.go in cmd, or move to tools/main.go and init new module
• install the tools via go install to a shared repo-local bin folder
• use the binaries from the bin folder instead of the go run construct in mage (via execGo)

Example Makefile (tools/Makefile):

GO = go
GOBIN = $(PWD)/.bin
export GOBIN

.PHONY: deps.dev
deps.dev:
	@cat main.go | grep _ | awk -F'"' '{print $$2}' | xargs -tI % $(GO) install %

And then shell exec tools like ./.bin/unconvert instead of constructing shell exec go run github.com/mdempsky/unconvert in mage and travis

[coveralls]

Coverage decreased (-0.002%) to 73.122% when pulling 49d17a60ceb38cdfac73fefe5190f5430d656ccd on feature/cmd-mod into c637d93 on master.

[johanstokking]

Should we turn this into $(GOBIN): tools/go.mod for it to track that file for changes to the tools and versions?

[rvolosatovs]

Why?

[rvolosatovs]

The benefit of using go run to run commands is the automatic installation/versioning with no Makefile etc. maintenance cost. What's the benefit of using compiled binaries? How do you propose to keep them up-to-date?

[johanstokking]

The benefit of using go run to run commands is the automatic installation/versioning with no Makefile etc. maintenance cost.

Yes, that definitely has benefits. But I do want to keep those modules (and replacements) out of the project root's go.mod. Two possible steps forward:

1. go run from the tools dir and account for that in the paths
2. Watch tools/go.mod in the Makefile so that changes trigger reinstallation

[rvolosatovs]

I'd vote for 1. We could move all the mage stuff there as well, in fact, I believe. That way we'd have everything tooling-related with all the dependency hell isolated in ./tools and a clean top-level module at root. I believe we don't need a ./cmd module either this way.

To clarify, that means running ./tools/mage instead of ./mage.

[johanstokking]

OK, please try if that works.

[johanstokking]

That way we'd have everything tooling-related with all the dependency hell isolated in ./tools and a clean top-level module at root

Looking at replacements we have in this changeset;

• gopkg.in/DATA-DOG/go-sqlmock.v1: that's because of github.com/heptiolabs/healthcheck required by pkg/component
• github.com/labstack/echo/v4: we're on an old version because of Valid web paths return 404 #977. With Starting point for migrating away from Echo #2382 we're closer to getting rid of Echo. @htdvisser can you file the issues to migrate away completely, as announced in that PR?
• github.com/Azure/go-autorest is quite challenging, see all: Azure go-autorest causing ambiguous import google/go-cloud#1990 (this is our dependency) and Azure go-autorest causing ambiguous import kubernetes/client-go#628
• github.com/robertkrimen/otto replaced by our fork: I keep pinging these guys (Use stack depth limit when exporting values robertkrimen/otto#283) but no luck there
• github.com/grpc-ecosystem/grpc-gateway: is this tooling?