build: Incomplete compile prerequisites and compile error related to yarn

[ghost]

Hello,

i just tried compiling the release-2.0 branch (rev. 5608e27) and noticed that yarn is missing in the prerequisites list at https://www.cockroachlabs.com/docs/stable/install-cockroachdb.html#build-from-source-linux

Also an error occurred when compiling:
build/node-run.sh -C ./pkg/ui yarn install
ERROR: [Errno 2] No such file or directory: 'install'
Makefile:322: recipe for target 'pkg/ui/yarn.installed' failed
make: *** [pkg/ui/yarn.installed] Error 1

I ran these commands
git clone -b release-2.0 https://github.com/cockroachdb/cockroach.git
cd cockroach
make build

[benesch]

Those prerequisites are for the source archive, which you can download here: https://binaries.cockroachdb.com/cockroach-v2.0.1.src.tgz

The source archive ships a precompiled version of the admin web UI, so you don't need Node and Yarn installed.

To build from our Git repository you'll need to adhere to the stricter list of prerequisites in CONTRIBUTING.md: https://github.com/cockroachdb/cockroach/blob/master/CONTRIBUTING.md

Apologies that this isn't more clear!

[ghost]

Sorry about that.

Thanks for the help, now it worked.

But maybe still one thing left: it seems go-bindata and goimports are not installed nor listed, yet required.
As I had to run:
go get -u github.com/a-urth/go-bindata/...
go get -u golang.org/x/tools/cmd/goimports
to successfully compile it.

Or did I overlooked that as well?

[benesch]

Sorry about that. Thanks for the help, now it worked.

No need to apologize! Appreciate you taking the time to report this issue.

You definitely should not need to manually install go-bindata nor goimports. We pin our dependencies using vendoring, then install them into the bin within the repository. That way we're not relying on whatever version of the tool you happen to have lying around in your PATH.

I'm not at all sure what went wrong in your case. You wouldn't happen to have the log from when you ran Make lying around, would you?

[ghost]

> uname -a
Linux builder 4.4.0-122-generic 146-Ubuntu SMP Mon Apr 23 15:34:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

> sudo apt-get update && 
> sudo apt-get -y install build-essential libc-dev libncurses-dev git bash make cmake autoconf

> cd && 
  git clone -b v6.14.2 https://github.com/nodejs/node.git node6.14.2 && 
  cd node6.14.2 && 
  ./configure 2>&1 | tee -a $HOME/node6.14.2.log && 
  make 2>&1 | tee -a $HOME/node6.14.2.log && 
  sudo make install 2>&1 | tee -a $HOME/node6.14.2.log

> cd && 
  sudo npm install -g yarn@1.0.2 2>&1 | tee -a $HOME/yarn1.0.2.log

> cd && 
  git clone -b release-branch.go1.4 https://go.googlesource.com/go go1.4 && 
  cd go1.4/src && 
  ./make.bash 2>&1 | tee -a $HOME/go1.4.log

> cd && 
  git clone -b release-branch.go1.10 https://go.googlesource.com/go go1.10 && 
  cd go1.10/src && 
  ./make.bash 2>&1 | tee -a $HOME/go1.10.log && 
  cd && 
  sudo mv -v go1.10 /usr/local/go

> export GOROOT=/usr/local/go && 
  export GOPATH=$HOME/go && 
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

> cd && 
  go get -v -d github.com/cockroachdb/cockroach && 
  cd $(go env GOPATH)/src/github.com/cockroachdb/cockroach && 
  make build 2>&1 | tee $HOME/build.log

> # ------------------------------
> cat $HOME/build.log

GOPATH set to /home/dummy/go
Running make with -j1
Running make with -j1
build/node-run.sh -C ./pkg/ui yarn install
yarn install v1.0.2
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info fsevents@1.1.1: The platform "linux" is incompatible with this module.
info "fsevents@1.1.1" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
warning "webpack-dev-middleware@1.10.1" has incorrect peer dependency "webpack@1 || ^2.1.0-beta || ^2.2.0-rc.0".
warning "ajv-keywords@2.1.0" has incorrect peer dependency "ajv@>=5.0.0".
[5/5] Building fresh packages...
Done in 11.17s.
# Prevent ProtobufJS from trying to install its own packages because a) the
# the feature is buggy, and b) it introduces an unnecessary dependency on NPM.
# Additionally pin a known-good version of jsdoc.
# See: https://github.com/dcodeIO/protobuf.js/issues/716.
cp ./pkg/ui/node_modules/protobufjs/cli/{package.standalone.json,package.json}
build/node-run.sh -C ./pkg/ui/node_modules/protobufjs/cli yarn add jsdoc@3.4.3
yarn add v1.0.2
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 65 new dependencies.
├─ acorn-jsx@3.0.1
├─ acorn@3.3.0
├─ align-text@0.1.4
├─ ansi-regex@2.1.1
├─ ansi-styles@2.2.1
├─ balanced-match@1.0.0
├─ bluebird@3.4.7
├─ brace-expansion@1.1.11
├─ camelcase@1.2.1
├─ catharsis@0.8.9
├─ center-align@0.1.3
├─ chalk@1.1.3
├─ cliui@2.1.0
├─ concat-map@0.0.1
├─ decamelize@1.2.0
├─ deep-is@0.1.3
├─ escape-string-regexp@1.0.5
├─ escodegen@1.9.1
├─ espree@3.5.4
├─ esprima@3.1.3
├─ estraverse@4.2.0
├─ esutils@2.0.2
├─ fast-levenshtein@2.0.6
├─ fs.realpath@1.0.0
├─ glob@7.1.2
├─ graceful-fs@4.1.11
├─ has-ansi@2.0.0
├─ inflight@1.0.6
├─ inherits@2.0.3
├─ is-buffer@1.1.6
├─ js2xmlparser@1.0.0
├─ jsdoc@3.4.3
├─ kind-of@3.2.2
├─ klaw@1.3.1
├─ lazy-cache@1.0.4
├─ levn@0.3.0
├─ longest@1.0.1
├─ marked@0.3.19
├─ minimatch@3.0.4
├─ minimist@1.2.0
├─ mkdirp@0.5.1
├─ once@1.4.0
├─ optionator@0.8.2
├─ os-tmpdir@1.0.2
├─ path-is-absolute@1.0.1
├─ prelude-ls@1.1.2
├─ repeat-string@1.6.1
├─ requizzle@0.2.1
├─ right-align@0.1.3
├─ semver@5.5.0
├─ source-map@0.5.7
├─ strip-ansi@3.0.1
├─ strip-json-comments@2.0.1
├─ supports-color@2.0.0
├─ taffydb@2.6.2
├─ tmp@0.0.31
├─ type-check@0.3.2
├─ uglify-js@2.8.29
├─ uglify-to-browserify@1.0.2
├─ underscore-contrib@0.3.0
├─ underscore@1.6.0
├─ window-size@0.1.0
├─ wordwrap@1.0.0
├─ wrappy@1.0.2
└─ yargs@3.10.0
Done in 1.88s.
build/node-run.sh -C ./pkg/ui/node_modules/protobufjs/cli yarn install
yarn install v1.0.2
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.16s.
rm -rf ./pkg/ui/node_modules/@types/node
touch pkg/ui/yarn.installed
# Add comment recognized by reviewable.
echo '// GENERATED FILE DO NOT EDIT' > pkg/ui/src/js/protos.js
build/node-run.sh ./pkg/ui/node_modules/.bin/pbjs -t static-module -w es6 --strict-long --keep-case --path ./pkg --path ./vendor/github.com/gogo/protobuf --path ./vendor/github.com/coreos --path ./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis ./pkg/server/serverpb/admin.proto ./pkg/server/serverpb/status.proto ./pkg/server/serverpb/authentication.proto ./pkg/ts/tspb/timeseries.proto >> pkg/ui/src/js/protos.js
# Add comment recognized by reviewable.
echo '// GENERATED FILE DO NOT EDIT' > pkg/ui/src/js/protos.d.ts
build/node-run.sh ./pkg/ui/node_modules/.bin/pbts ./pkg/ui/src/js/protos.js >> pkg/ui/src/js/protos.d.ts
build/node-run.sh -C ./pkg/ui ./node_modules/.bin/webpack -p --config webpack.protos.js
[BABEL] Note: The code generator has deoptimised the styling of "/home/dummy/go/src/github.com/cockroachdb/cockroach/pkg/ui/src/js/protos.js" as it exceeds the max of "500KB".
Hash: 3091ba5fcc40d1287043
Version: webpack 3.8.1
Time: 17455ms
        Asset     Size  Chunks                    Chunk Names
protos.dll.js  1.19 MB       0  [emitted]  [big]  protos
  [25] dll protos 12 bytes {0} [built]
  [26] ./src/js/protos.js 3.09 MB {0} [built]
  [31] (webpack)/buildin/global.js 823 bytes {0} [built]
    + 43 hidden modules
build/node-run.sh -C ./pkg/ui ./node_modules/.bin/webpack -p --config webpack.vendor.js
Hash: 551015e4a1d3899862cc
Version: webpack 3.8.1
Time: 15698ms
        Asset     Size  Chunks                    Chunk Names
vendor.dll.js  1.92 MB       0  [emitted]  [big]  vendor
   [8] (webpack)/buildin/global.js 488 bytes {0} [built]
  [51] (webpack)/buildin/module.js 495 bytes {0} [built]
 [306] dll vendor 12 bytes {0} [built]
 [709] ./node_modules/moment/locale ^\.\/.*$ 2.79 kB {0} [optional] [built]
    + 759 hidden modules
rm -f ./pkg/ui/embedded.go
find ./pkg/ui/distccl -mindepth 1 -not -name distccl.go -delete
set -e; for dll in protos.dll.js vendor.dll.js; do ln -s ../dist/$dll ./pkg/ui/distccl/$dll; done
build/node-run.sh -C ./pkg/ui ./node_modules/.bin/webpack --config webpack.ccl.js
ts-loader: Using typescript@2.6.2 and /home/dummy/go/src/github.com/cockroachdb/cockroach/pkg/ui/tsconfig.json
Hash: c9957acee5263a575607
Version: webpack 3.8.1
Time: 14092ms
                                 Asset       Size  Chunks                    Chunk Names
bd03a2cc277bbbc338d464e679fe9942.woff2     183 kB          [emitted]
  554f370d0fd6e0b022c36689cfd8190b.gif    35.8 kB          [emitted]
c9cbbdc3762c340d5d37073a54971487.woff2     185 kB          [emitted]
 2ce4d82354fdf1be1788c526d94eefc1.woff     310 kB          [emitted]  [big]
cccb897485813c7c256901dbca54ecf2.woff2     185 kB          [emitted]
 d878b6c29b10beca227e9eef4246111b.woff     310 kB          [emitted]  [big]
0996d39c4cf5d223a14559dfa37047fd.woff2     182 kB          [emitted]
 acbd6ecc97c80340e9bf00ea80063234.woff     306 kB          [emitted]  [big]
  22230dddbcc6fe6fd41e540687e9c68c.png    94.3 kB          [emitted]
 27bd77b9162d388cb8d4c4217c7c5e2a.woff     309 kB          [emitted]  [big]
 5d4981a99e52bb068c3d6d1311b134b6.woff    12.6 kB          [emitted]
  9a7de44c0fd4d701d61396ee25408bb6.svg    14.5 kB          [emitted]
  48fc9d04ca5b94535f9b8eacaeff0dff.svg    75.2 kB          [emitted]
                             bundle.js    2.61 MB       0  [emitted]  [big]  main
                           favicon.ico    33.3 kB          [emitted]
                            index.html  958 bytes          [emitted]
   [0] delegated ./node_modules/react/index.js from dll-reference vendor_551015e4a1d3899862cc 42 bytes {0} [built]
   [7] delegated ./node_modules/react-router/es/index.js from dll-reference vendor_551015e4a1d3899862cc 42 bytes {0} [built]
 [103] multi ./src/index.tsx 28 bytes {0} [built]
 [104] ./src/index.tsx 12 kB {0} [built]
 [266] ./src/views/jobs/index.tsx 16.7 kB {0} [built]
 [273] ./src/views/reports/containers/certificates/index.tsx 13.2 kB {0} [built]
 [274] ./src/views/reports/containers/commandQueue/index.tsx 8.76 kB {0} [built]
 [289] ./src/views/reports/containers/customChart/index.tsx 15.6 kB {0} [built]
 [293] ./src/views/reports/containers/debug/index.tsx 12 kB {0} [built]
 [296] ./src/views/reports/containers/problemRanges/index.tsx 11.5 kB {0} [built]
 [298] ./src/views/reports/containers/localities/index.tsx 8.82 kB {0} [built]
 [301] ./src/views/reports/containers/network/index.tsx 23.3 kB {0} [built]
 [302] ./src/views/reports/containers/nodes/index.tsx 15 kB {0} [built]
 [303] ./src/views/reports/containers/redux/index.tsx 4.31 kB {0} [built]
 [307] ./src/views/reports/containers/range/index.tsx 10.6 kB {0} [built]
    + 299 hidden modules
Child html-webpack-plugin for "index.html":
     1 asset
       [2] (webpack)/buildin/global.js 488 bytes {0} [built]
       [3] (webpack)/buildin/module.js 495 bytes {0} [built]
        + 2 hidden modules
go-bindata -pkg distccl -o pkg/ui/distccl/bindata.go -prefix ./pkg/ui/distccl ./pkg/ui/distccl/...
bash: go-bindata: command not found
Makefile:1125: recipe for target 'pkg/ui/distccl/bindata.go' failed
make: *** [pkg/ui/distccl/bindata.go] Error 127

[benesch]

Hmm, thanks for sending that along. It's not clear to me if Make is failing to set PATH appropriately or if it failed to install go-bindata into the bin directory. If you run rm -rf bin, does the problem go away? If so, that's evidence for the latter theory (failed installation). If not, it's evidence for the former theory (bad PATH).

[ghost]

Hey @benesch, sorry for the late answer.
Actually I simply created it from scratch again, however it still fails.

Running this
go get -v -u github.com/a-urth/go-bindata/...
solves it.

I also noticed that the dependencies of cockroach are resolved later (after the failing go-bindata call)

As the second build.log contains a pull from a different go-bindata repo (github.com/jteeuwen/go-bindata)

failing build log:
https://gist.github.com/Dendon/67c84c3ee8a81ca20a1050015f0bce98

successful build log:
https://gist.github.com/Dendon/8a59e42cd27c4a6009e023f711630e37

[rendaw]

Not sure if this helps, but I am able to reproduce this. I'm building in a Dockerfile and while it works locally for me it fails in Docker hub. The only difference between the two I can see is the number of cpus - in hub it's limited to 1 cpu, locally I see the build uses -j15 or something.

To confirm this I ran the docker build locally with --cpuset-cpus 1 and encountered the failure. And I just made another build without cpuset again and it worked.

[benesch]

Thanks for your patience, @Dendon and @rendaw—and thanks for reporting! I finally tracked down the issue. The fix, in #27642, will land later today.