Scoped package installation sometimes fails on Windows

[dpoindexter]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
When a project depends on a scoped package on Windows, the package linking step fails 50% of the time if any subdependencies of the scoped package aren't hoisted out of the scoped package.

If the current behavior is a bug, please provide the steps to reproduce.

1. yarn add a scoped package (ex. @cycle/http)
2. yarn add any dependency of the scoped package, at an incompatible version (ex. superagent@1.7.0) (This prevents the subdependency of the scoped package from hoisting)
3. Run the same yarn add a second time
4. Error on step 3

What is the expected behavior?
Running yarn add again for the same package should successfully complete and re-link packages

Please mention your node.js, yarn and operating system version.
node: v6.9.1
yarn: v0.17.0
os: Windows 10

yarn-error.log

Arguments: C:\Program Files\nodejs\node.exe C:\Users\dpoindexter\AppData\Roaming\npm\node_modules\yarn\bin\yarn.js add superagent@1.7.0

PATH: 
  C:\Users\dpoindexter\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\local\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\bin;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\dpoindexter\bin;C:\cmder\vendor\conemu-maximus5\ConEmu\Scripts;C:\cmder\vendor\conemu-maximus5;C:\cmder\vendor\conemu-maximus5\ConEmu;C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0;C:\ProgramData\chocolatey\bin;C:\Program Files\TortoiseHg;C:\Program Files\Git\cmd;C:\Program Files\Microsoft SQL Server\110\Tools\Binn;C:\Program Files\Microsoft SQL Server\120\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn\ManagementStudio;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.0;C:\Program Files\nodejs;C:\Users\dpoindexter\AppData\Local\atom\bin;C:\Program Files\Sublime Text 3;C:\Users\dpoindexter\AppData\Roaming\npm;C:\Program Files\7-Zip;C:\Program Files\Git\usr\bin\vendor_perl;C:\Program Files\Git\usr\bin\core_perl

Yarn version: 
  0.17.0

Node version: 
  6.9.1

Platform: 
  win32 x64

npm manifest: 
  {
    "name": "test2",
    "version": "1.0.0",
    "main": "index.js",
    "license": "MIT",
    "dependencies": {
      "@cycle/http": "^11.2.0",
      "superagent": "1.7.0"
    }
  }

yarn manifest: 
  No manifest

Lockfile: 
  # THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
  # yarn lockfile v1
  
  
  "@cycle/http@^11.2.0":
    version "11.2.0"
    resolved "https://registry.yarnpkg.com/@cycle/http/-/http-11.2.0.tgz#fd33ca972aa685e261567d7dd26692102b849ab8"
    dependencies:
      "@cycle/xstream-adapter" "3.x.x"
      superagent "2.3.0"
  
  "@cycle/xstream-adapter@3.x.x":
    version "3.0.4"
    resolved "https://registry.yarnpkg.com/@cycle/xstream-adapter/-/xstream-adapter-3.0.4.tgz#54c440b02ef3c8433cdb2d31082c8e5d0dd4d93a"
  
  async@^1.5.2:
    version "1.5.2"
    resolved "https://registry.yarnpkg.com/async/-/async-1.5.2.tgz#ec6a61ae56480c0c3cb241c95618e20892f9672a"
  
  async@~0.9.0:
    version "0.9.2"
    resolved "https://registry.yarnpkg.com/async/-/async-0.9.2.tgz#aea74d5e61c1f899613bf64bda66d4c78f2fd17d"
  
  buffer-shims@^1.0.0:
    version "1.0.0"
    resolved "https://registry.yarnpkg.com/buffer-shims/-/buffer-shims-1.0.0.tgz#9978ce317388c649ad8793028c3477ef044a8b51"
  
  combined-stream@^1.0.5:
    version "1.0.5"
    resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.5.tgz#938370a57b4a51dea2c77c15d5c5fdf895164009"
    dependencies:
      delayed-stream "~1.0.0"
  
  combined-stream@~0.0.4:
    version "0.0.7"
    resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-0.0.7.tgz#0137e657baa5a7541c57ac37ac5fc07d73b4dc1f"
    dependencies:
      delayed-stream "0.0.5"
  
  component-emitter@^1.2.0, component-emitter@~1.2.0:
    version "1.2.1"
    resolved "https://registry.yarnpkg.com/component-emitter/-/component-emitter-1.2.1.tgz#137918d6d78283f7df7a6b7c5a63e140e69425e6"
  
  cookiejar@2.0.6:
    version "2.0.6"
    resolved "https://registry.yarnpkg.com/cookiejar/-/cookiejar-2.0.6.tgz#0abf356ad00d1c5a219d88d44518046dd026acfe"
  
  cookiejar@^2.0.6:
    version "2.1.0"
    resolved "https://registry.yarnpkg.com/cookiejar/-/cookiejar-2.1.0.tgz#86549689539b6d0e269b6637a304be508194d898"
  
  core-util-is@~1.0.0:
    version "1.0.2"
    resolved "https://registry.yarnpkg.com/core-util-is/-/core-util-is-1.0.2.tgz#b5fd54220aa2bc5ab57aab7140c940754503c1a7"
  
  debug@2, debug@^2.2.0:
    version "2.3.2"
    resolved "https://registry.yarnpkg.com/debug/-/debug-2.3.2.tgz#94cb466ef7d6d2c7e5245cdd6e4104f2d0d70d30"
    dependencies:
      ms "0.7.2"
  
  delayed-stream@0.0.5:
    version "0.0.5"
    resolved "https://registry.yarnpkg.com/delayed-stream/-/delayed-stream-0.0.5.tgz#d4b1f43a93e8296dfe02694f4680bc37a313c73f"
  
  delayed-stream@~1.0.0:
    version "1.0.0"
    resolved "https://registry.yarnpkg.com/delayed-stream/-/delayed-stream-1.0.0.tgz#df3ae199acadfb7d440aaae0b29e2272b24ec619"
  
  extend@1.2.1:
    version "1.2.1"
    resolved "https://registry.yarnpkg.com/extend/-/extend-1.2.1.tgz#a0f5fd6cfc83a5fe49ef698d60ec8a624dd4576c"
  
  extend@^3.0.0:
    version "3.0.0"
    resolved "https://registry.yarnpkg.com/extend/-/extend-3.0.0.tgz#5a474353b9f3353ddd8176dfd37b91c83a46f1d4"
  
  form-data@0.2.0:
    version "0.2.0"
    resolved "https://registry.yarnpkg.com/form-data/-/form-data-0.2.0.tgz#26f8bc26da6440e299cbdcfb69035c4f77a6e466"
    dependencies:
      async "~0.9.0"
      combined-stream "~0.0.4"
      mime-types "~2.0.3"
  
  form-data@1.0.0-rc4:
    version "1.0.0-rc4"
    resolved "https://registry.yarnpkg.com/form-data/-/form-data-1.0.0-rc4.tgz#05ac6bc22227b43e4461f488161554699d4f8b5e"
    dependencies:
      async "^1.5.2"
      combined-stream "^1.0.5"
      mime-types "^2.1.10"
  
  formidable@^1.0.17, formidable@~1.0.14:
    version "1.0.17"
    resolved "https://registry.yarnpkg.com/formidable/-/formidable-1.0.17.tgz#ef5491490f9433b705faa77249c99029ae348559"
  
  inherits@~2.0.1:
    version "2.0.3"
    resolved "https://registry.yarnpkg.com/inherits/-/inherits-2.0.3.tgz#633c2c83e3da42a502f52466022480f4208261de"
  
  isarray@0.0.1:
    version "0.0.1"
    resolved "https://registry.yarnpkg.com/isarray/-/isarray-0.0.1.tgz#8a18acfca9a8f4177e09abfc6038939b05d1eedf"
  
  isarray@~1.0.0:
    version "1.0.0"
    resolved "https://registry.yarnpkg.com/isarray/-/isarray-1.0.0.tgz#bb935d48582cba168c06834957a54a3e07124f11"
  
  methods@^1.1.1, methods@~1.1.1:
    version "1.1.2"
    resolved "https://registry.yarnpkg.com/methods/-/methods-1.1.2.tgz#5529a4d67654134edcc5266656835b0f851afcee"
  
  mime-db@~1.12.0:
    version "1.12.0"
    resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.12.0.tgz#3d0c63180f458eb10d325aaa37d7c58ae312e9d7"
  
  mime-db@~1.24.0:
    version "1.24.0"
    resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.24.0.tgz#e2d13f939f0016c6e4e9ad25a8652f126c467f0c"
  
  mime-types@^2.1.10:
    version "2.1.12"
    resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.12.tgz#152ba256777020dd4663f54c2e7bc26381e71729"
    dependencies:
      mime-db "~1.24.0"
  
  mime-types@~2.0.3:
    version "2.0.14"
    resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.0.14.tgz#310e159db23e077f8bb22b748dabfa4957140aa6"
    dependencies:
      mime-db "~1.12.0"
  
  mime@1.3.4, mime@^1.3.4:
    version "1.3.4"
    resolved "https://registry.yarnpkg.com/mime/-/mime-1.3.4.tgz#115f9e3b6b3daf2959983cb38f149a2d40eb5d53"
  
  ms@0.7.2:
    version "0.7.2"
    resolved "https://registry.yarnpkg.com/ms/-/ms-0.7.2.tgz#ae25cf2512b3885a1d95d7f037868d8431124765"
  
  process-nextick-args@~1.0.6:
    version "1.0.7"
    resolved "https://registry.yarnpkg.com/process-nextick-args/-/process-nextick-args-1.0.7.tgz#150e20b756590ad3f91093f25a4f2ad8bff30ba3"
  
  qs@2.3.3:
    version "2.3.3"
    resolved "https://registry.yarnpkg.com/qs/-/qs-2.3.3.tgz#e9e85adbe75da0bbe4c8e0476a086290f863b404"
  
  qs@^6.1.0:
    version "6.3.0"
    resolved "https://registry.yarnpkg.com/qs/-/qs-6.3.0.tgz#f403b264f23bc01228c74131b407f18d5ea5d442"
  
  readable-stream@1.0.27-1:
    version "1.0.27-1"
    resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-1.0.27-1.tgz#6b67983c20357cefd07f0165001a16d710d91078"
    dependencies:
      core-util-is "~1.0.0"
      inherits "~2.0.1"
      isarray "0.0.1"
      string_decoder "~0.10.x"
  
  readable-stream@^2.0.5:
    version "2.2.2"
    resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-2.2.2.tgz#a9e6fec3c7dda85f8bb1b3ba7028604556fc825e"
    dependencies:
      buffer-shims "^1.0.0"
      core-util-is "~1.0.0"
      inherits "~2.0.1"
      isarray "~1.0.0"
      process-nextick-args "~1.0.6"
      string_decoder "~0.10.x"
      util-deprecate "~1.0.1"
  
  reduce-component@1.0.1:
    version "1.0.1"
    resolved "https://registry.yarnpkg.com/reduce-component/-/reduce-component-1.0.1.tgz#e0c93542c574521bea13df0f9488ed82ab77c5da"
  
  string_decoder@~0.10.x:
    version "0.10.31"
    resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-0.10.31.tgz#62e203bc41766c6c28c9fc84301dab1c5310fa94"
  
  superagent@1.7.0:
    version "1.7.0"
    resolved "https://registry.yarnpkg.com/superagent/-/superagent-1.7.0.tgz#fa40411e78525b74fbf9cde8e4c0afc43af851e5"
    dependencies:
      component-emitter "~1.2.0"
      cookiejar "2.0.6"
      debug "2"
      extend "1.2.1"
      form-data "0.2.0"
      formidable "~1.0.14"
      methods "~1.1.1"
      mime "1.3.4"
      qs "2.3.3"
      readable-stream "1.0.27-1"
      reduce-component "1.0.1"
  
  superagent@2.3.0:
    version "2.3.0"
    resolved "https://registry.yarnpkg.com/superagent/-/superagent-2.3.0.tgz#703529a0714e57e123959ddefbce193b2e50d115"
    dependencies:
      component-emitter "^1.2.0"
      cookiejar "^2.0.6"
      debug "^2.2.0"
      extend "^3.0.0"
      form-data "1.0.0-rc4"
      formidable "^1.0.17"
      methods "^1.1.1"
      mime "^1.3.4"
      qs "^6.1.0"
      readable-stream "^2.0.5"
  
  util-deprecate@~1.0.1:
    version "1.0.2"
    resolved "https://registry.yarnpkg.com/util-deprecate/-/util-deprecate-1.0.2.tgz#450d4dc9fa70de732762fbd2d4a28981419a0ccf"

Trace: 
  Error: ENOENT: no such file or directory, open 'C:\development\test2\node_modules\@cycle\http\node_modules\isarray\.npmignore'
      at Error (native)

[maybeec]

Confirm: Running 0.17.2 the issue of 0.16.1 persists. Getting

Trace: 
  Error: ENOENT: no such file or directory, open 'XXXXX\node_modules\@types\lodash\index.d.ts'
      at Error (native)

on windows 7 x64 / node 6.9.1

Running yarn add a second time, it works.

[maybeec]

persists with 0.17.3

Trace: 
  Error: ENOENT: no such file or directory, open 'XXXXX\node_modules\@types\jasmine\index.d.ts'
      at Error (native)

[ecozoic]

Also seeing it with 0.17.4

[dpoindexter]

@maybeec, @ecozoic: If this is blocking you at the moment, it's possible to work around if you have some control over your dependency tree. There's a complete description of the underlying problem on this PR, but the TL;DR is that all subdependencies of the scoped dependency must be hoistable. So, the scoped dep can't share any semver-incompatible subdependencies with the rest of the tree. --flat might be one way to resolve this.

[maybeec]

After working on some other workarounds of yarn bugs, I came back to this issue.
@dpoindexter Your pull request did not solve the issue entirely as far as I understood it, right?
Basically, I am still observing the issue. The issue could be resolved by using --flat, but I am not convinced that this issue can be closed entirely or we should open a new one, right? I could not finde any open issue regarding this.

Unfortunately, I currently also cannot run a successfull build of yarn --flat due to the current fsevents issues. Btw. just also got a ENOENT for @types packages on yarn --flat which seems to be again another issue on clean installation.

[maybeec]

ok, after some debugging, I found the same things @dpoindexter already fixed.
Then I recognized, that a lot issues like also this one have just been resolved on master branch rather than on 0.17 bug fix branch. Might it be possible to get a cherry pick for that? I would love to see this @bestander.

[bestander]

Considering that branches diverged too much let's make a 0.18 release later this week

[dpoindexter]

@bestander That would be excellent. This is the last remaining issue preventing us from moving fully to yarn.

[maybeec]

That sounds nice, I would love to see this release this week as it would be best fitting to my upcoming rollout next week.
However, I also observed the regression bug #2051, which might be a new blocker if not fixed in 0.18.x. I know, this is much demanded of your team. But it would help to rollout and accept yarn, which in my optinion is a great step further.