Support for Yarn package manager

[amasad]

Yarn is a new package manager -- an npm replacement -- by Facebook and Google. It's faster, more reliable and secure than npm. I'd like to use it to manage my Heroku node app dependencies.

[hunterloftis]

So would I. This feature will be coming soon.

[rstacruz]

👍

If I may suggest, maybe an ENV config variable like USE_YARN=1 would do.

[kane-thornwyrd]

It's faster, more reliable and secure than npm

Fact please.

[amasad]

@kane-thornwyrd the website has tons of facts for you: https://yarnpkg.com/

[rstacruz]

Oh, and if I may suggest again, the presence of a yarn.lock file should automatically use yarn for you. 😊

[intellix]

@kane-thornwyrd

Performance: https://yarnpkg.com/en/compare (try it locally, results are insane)
Reliable: The results are deterministic. The amount of times I've had to tell members of my team to: "it worked on my machine... try again maybe? great, it worked this time"

[kane-thornwyrd]

Sorry everyone if it may sound like defiance, I think that's a good thing to add Yarn to Heroku, and anyway my opinion does not count about this, and sorry to clutter this demand, I just seek some rock solid facts about yarn among its early supporters.

@intellix thank you for your answers, I have more indepth questions:

Performances Without a proper protocol, checking which servers are requested, the network quality during the tests and some other data, I'm not particullary receptive to this kind of graph.
Moreover, in my case, given I don't consider the re-/installation like something done often, perfs here are a margin interest. But, okay, that's a cool thing to have anyway.

Reliable In 3 years the only package manager related issue I witnessed, among my heteroclite teams (Ubuntu, Fedora, Win7, Win10,Debian over SSH on Win7, OS X...), was all the binary dependencies requiring node-gyp/Visual Studio/XCode/etc.
What you're decribing is quite strange, I mean, you should have worked on ultra complex app with dependencies on multiple versions of the same libraries, and in this case the Flat mode which:

Resolve mismatching versions of dependencies to a single version to avoid creating duplicates

Would ruin your day, am I wrong ?

more Secure ?

[technomage]

More secure because it records checksums of installed packages on update and verifies on install. This prevents a man in the middle from substituting a rogue package.

[technomage]

Typical npm version issues result from loose package.json (*, >=, ^) that result in different versions when installed at different times. In teams that religiously npm install after each pull and clear out old versions, this may not be an issue.

[kane-thornwyrd]

@technomage : thank you for your answer, I don't get how yarn is supposed to authenticate packages since it is supposed to support custom repos, then I guess it use the sha-1 digest to ensure a proper download, as npm. That does not prevent a MitM attack. Does it store the hashes ? Then how it is supposed to be aware of the new ones from a minor release ?

[hunterloftis]

This discussion is getting into the weeds. We at Heroku, and more importantly our users, feel the same pains with npm as described in Facebook's announcement. We're very excited about Yarn and will be adding support for it (I'm working on that right at this moment). Everything has tradeoffs, but there are better places to discuss them :)

[hunterloftis]

Give it a shot:

heroku buildpacks:set https://github.com/heroku/heroku-buildpack-nodejs#yarn

[lfittl]

@hunterloftis Works well here - thanks for the super-fast turnaround on this :)

[parse]

@hunterloftis The postinstall script doesnt seem to be executed on Heroku using the build pack above. It looks like it is fixed in yarnpkg/yarn#800, do you think its related?

Side note: On CircleCI (our continuous integration software) we had to specify '- yarn run postinstall' in our post declarations.

[drewhamlett]

Only fact needed is to know Yehuda Katz (Bundler, Cargo) is involved. Thanks for your help

[joshmanders]

Any status on this being merged in and a new version cut so that I can put it into my build system?

[amasad]

@hunterloftis this works well! Only issue, as @parse mentioned this doesn't run postinstall. However, I was able to get around this by using heroku-postbuild custom script hook.

[voldern]

@hunterloftis this works well! Only issue, as @parse mentioned this doesn't run postinstall.

This has been fixed in v0.16.0 that was released a day ago. See yarnpkg/yarn#800

[steffenagger]

Yep, this works @hunterloftis. Maybe include a version check looking at engines.yarn?

[mhuebert]

@hunterloftis thanks for getting this out so fast!

I've discovered a problem related to caching, and pushed a minimal example with steps to reproduce the issue: https://github.com/mhuebert/heroku-yarn-test

The gist: if a project depends on "bcrypt", the first build/heroku-push will work, but subsequent builds (after changing dependencies) will fail with an error:

/app/node_modules/bindings/bindings.js:91
  ^

Error: Could not locate the bindings file. Tried:
 → /app/node_modules/bcrypt/build/Debug/bcrypt_lib.node
 → /app/node_modules/bcrypt/build/bcrypt_lib.node
 → /app/node_modules/bcrypt/build/Release/bcrypt_lib.node
 → /app/node_modules/bcrypt/out/Debug/bcrypt_lib.node
 → /app/node_modules/bcrypt/Debug/bcrypt_lib.node

After this happens, the only way I can get any subsequent build to work is by using heroku-repo to purge the cache (heroku repo:purge_cache)

I do see a recent related issue on yarn, however I don't see this error when I run locally (and I am also on Sierra), nor on CircleCi.

[Twipped]

I've been running into the bcrypt problem as well on my codeship builds, so I don't think it's a heroku problem. Something is wrong with the way yarn replaces the bindings when it re-runs install scripts. I had to start deleting my node_modules folder before every build.

[hunterloftis]

This will disable the cache (for modules like bcrypt that don't yet work well with yarn's cache):

$ heroku config:set NODE_MODULES_CACHE=false

[tillkruss]

I've set:

heroku buildpacks:set https://github.com/heroku/heroku-buildpack-nodejs#yarn

But the deploy fails. Any tips?

-----> Node.js app detected
-----> Creating runtime environment

       NPM_CONFIG_LOGLEVEL=error
       NPM_CONFIG_PRODUCTION=true
       NODE_ENV=production
       NODE_MODULES_CACHE=true
-----> Installing binaries
       engines.node (package.json):  unspecified
       engines.npm (package.json):   unspecified (use default)

       Resolving node version (latest stable) via semver.io...
       Downloading and installing node 5.11.1...
       Using default npm version: 3.8.6
-----> Restoring cache
       Loading 2 from cacheDirectories (default):
       - node_modules
       - bower_components (not cached - skipping)
-----> Building dependencies
       Installing node modules (package.json)
-----> Caching build
       Clearing previous node cache
       Saving 2 cacheDirectories (default):
       - node_modules
       - bower_components (nothing to cache)
-----> Build succeeded!
       └── (empty)

-----> PHP app detected
-----> Bootstrapping...
-----> Installing platform packages...
[...snip...]
-----> Installing dependencies...
       Composer version 1.2.1 2016-09-12 11:27:19
       Loading composer repositories with package information
       Installing dependencies from lock file
[...snip...]
-----> Running 'composer compile'...
       > (cd web/app/themes/caroline && yarn)
       sh: 1: yarn: not found
       Script (cd web/app/themes/caroline && yarn) handling the compile event returned with error code 127
 !     Push rejected, failed to compile PHP app.
 !     Push failed

[natecox]

@kane-thornwyrd here are practical CI benchmarks for yarn, if these haven't already been linked somewhere above: https://medium.freecodecamp.com/npm-vs-yarn-benchmark-9b456de4aa96#.sa32q3heg

[runlevel5]

@kane-thornwyrd I ran into more issues with npm such as sometimes the npm failed to fetch, errors out with 404. Plus npm runs slow as hell. That alone gives enough push for yarn in Heroku. I am not jumping on the hype bandwagon, but I have to admit npm does not deliver the reliability that users deserve.

[kane-thornwyrd]

@joneslee85 as @hunterloftis said, there is other places to discuss this topic.

[AndrewBarba]

This is my package.json, not exactly anything crazy:

{
  "name": "...",
  "description": "...",
  "author": "...",
  "private": true,
  "engines": {
    "node": "6.9.2"
  },
  "scripts": {
    "start": "node server"
  },
  "dependencies": {
    "bluebird": "3.4.6",
    "compression": "1.6.2",
    "ejs": "2.5.5",
    "express": "4.14.0",
    "forky": "0.3.0",
    "prerender-node": "2.6.0",
    "serve-static": "1.11.1"
  }
}

[natecox]

@AndrewBarba I think I missed something here. Isn't the issue that you don't want dev dependencies installed on production? From your package.json it doesn't look like any of your dependencies are marked as dev only.

[AndrewBarba]

@natecox Someone else made that comment, in a previous comment I said that we didn't have any dev dependencies and it is still breaking for us. First person to mention there was an issue said that a config var was being ignored, but that is an open issue in Yarn. For me, i Just want production dependencies, and on a clean installation of yarn, no node modules folder, running yarn works correctly locally. It doesn't on Heroku

Previous comment for reference, sorry for confusion!

We don't set that flag, don't want any dev dependencies, and Yarn still broke for us. Our package.json only has 5 production deps on the app I posted above

[natecox]

Ah, got it.

[hunterloftis]

@AndrewBarba yarn relies on a lockfile. What does yours look like?

[AndrewBarba]

@hunterloftis Yes, I'm aware. Thats why this is so concerning. The whole point of Yarn is to install exactly what is in the lock file. The same one checked into source, on all dev machines, and Heroku. When everything runs locally, all CI passes, Heroku build passes, server goes to start and THAT's where it breaks down... not fun. If build breaks, fine. But thinking the build/install was successful and deploying to all dynos? Terrifying.

Here is lock: https://assets.tablelist.com/fyre/yarn.lock.zip

[AndrewBarba]

Okay digging into actual build logs, this is what a successful looks like (was able to get this to succeed just now on a dev app with NODE_ENV=development):

-----> Building dependencies
       Installing node modules (yarn)
       yarn install v0.17.10
       [1/4] Resolving packages...
       warning gulp > vinyl-fs > glob-stream > minimatch@2.0.10: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
       warning gulp > vinyl-fs > glob-watcher > gaze > globule > minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
       warning gulp > vinyl-fs > glob-watcher > gaze > globule > glob > graceful-fs@1.2.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
       success Already up-to-date.
       Done in 3.09s.
-----> Caching build
       Clearing previous node cache
       Saving 2 cacheDirectories (default):
       - node_modules
       - bower_components (nothing to cache)
-----> Build succeeded!
...
├─ async@0.2.10
       ├─ asynckit@0.4.0
       ├─ autoprefixer@6.5.4
       │  ├─ browserslist@~1.4.0
       │  ├─ caniuse-db@^1.0.30000597
       │  ├─ normalize-range@^0.1.2
       │  ├─ num2fraction@^1.2.2
       │  ├─ postcss-value-parser@^3.2.3
       │  └─ postcss@^5.2.6

And this is what the build looked like when the app couldn't start:

-----> Building dependencies
       Installing node modules (yarn)
       yarn install v0.17.10
       [1/4] Resolving packages...
       success Already up-to-date.
       Done in 0.57s.
-----> Caching build
       Clearing previous node cache
       Saving 2 cacheDirectories (default):
       - node_modules
       - bower_components (nothing to cache)
-----> Build succeeded!
...
?????? async@0.2.10
       ?????? asynckit@0.4.0
       ?????? autoprefixer@6.5.4
       ???  ?????? browserslist@~1.4.0
       ???  ?????? caniuse-db@^1.0.30000597
       ???  ?????? normalize-range@^0.1.2
       ???  ?????? num2fraction@^1.2.2
       ???  ?????? postcss-value-parser@^3.2.3
       ???  ?????? postcss@^5.2.6

So something strange going on between the two builds. Almost like it thought that it cached deps on my production app

[AndrewBarba]

I was just able to deploy to my production app with NODE_MODULES_CACHE=false, same yarn.lock file as before

[adamreisnz]

I guess I spoke to soon. Getting Error: Cannot find module '/app/node_modules/bcrypt/lib/binding/bcrypt_lib.node' error now in staging environment. Will try to disable NODE_MODULES_CACHE and see if that helps.

Edit: yes that worked.

[adamreisnz]

I think in light of all the issues people are experiencing, maybe the build pack should not automatically use yarn if there is a yarn.lock file present, but rather work off what is specified in package.json. Maybe if npm is specified as an engine, use npm. If yarn is, then use yarn. Or some other Heroku specific setting..

[gordonk]

@hunterloftis I was unable to deploy my node/yarn application and setting the build pack back to pre-yarn was only solution mentioned here that worked. Otherwise crashed when trying to start the express server.

2016-12-17T08:35:17.436016+00:00 app[web.1]: Error: Cannot find module 'mime'
2016-12-17T08:35:17.436017+00:00 app[web.1]:     at Function.Module._resolveFilename (module.js:469:15)

Npm install worked fine.

[adamreisnz]

We just had to remove the yarn.lock file from our project because the build was failing in production 😢

I guess we'll wait a bit longer until yarn is more stable.

[drewhamlett]

Are you using the master branch? We were using yarn branch but it seems the master branch supports yarn now. It's been working for us.

[adamreisnz]

Yes, master branch

[pasupulaphani]

yeah and it does respect yarn version put under engines

  "engines": {
    "node": "7.0.x",
    "yarn": "0.16.1"
  },

[tribou]

For anyone else searching, I found a couple more workarounds until this issue gets fixed:

• Add yarn.lock to a .slugignore file so Heroku goes back to using NPM for now. (solution I chose)

OR

• Set your buildpack to the last working version with heroku buildpacks:set https://github.com/heroku/heroku-buildpack-nodejs#v91

[robinvdvleuten]

Same here! Build is failing because the app requires any private modules...maybe a warning ahead by Heroku was in place here. A relative new tool like Yarn still has a lot of rough edges, maybe wait for a couple of months to merge such thing in a production platform or make it a beta feature for users who want to run their app on the wild side of things.

[joshmanders]

@robinvdvleuten you can opt out of using yarn by adding yarn.lock to your .slugignore file. Putting Yarn into this buildpack is one of the best ways to help the Yarn team find all edge cases and get them fixed, because they have a higher sample of users to get info from.

[freeall]

I'm guessing Heroku didn't know that yarn doesn't support private github modules - which quite a lot of people use

[hunterloftis]

@freeall @robinvdvleuten yes, we're aware that yarn doesn't support private modules. Why are you using yarn / have a yarn.lock file in the root of your project, if you are relying on functionality that yarn doesn't yet support? Yarn is only used in the buildpack if you check yarn.lock into git.

Yarn is a new technology, and while it's building the vast majority of apps faster and with determinism, it may not be suited for your project. If you want to have a yarn.lock file, but do not want to use yarn (?), I've added instructions for opting out here:

• https://devcenter.heroku.com/articles/nodejs-support#build-behavior

[hunterloftis]

Yarn updates:

#359

[joeyespo]

Looks like there's another problem when a devDependencies includes binaries (when NODE_ENV=production and NPM_CONFIG_PRODUCTION=false) yarnpkg/yarn#2214

[nascVi]

Can't say that much about Yarn! Thanks landscapersParis is also on slack as known as geeks gardens and landscapers

[fozzle]

It seems to me that if I use the default cache settings, any packages with native bindings will fail after the first deploy.

Disabling the module cache fixes this (as many have noted) but I'm curious if anyone has figured out a way to make the cache play nicely with native bindings now that it's been a few months.