Building apps in parallel fails

[wimme]

Current Behavior

Since upgrading our monorepo from Nx 14.5.10 to 15.9.2 we experience build issues on our CI build pipeline:

First it builds all Angular publishable libs sequentially by taking dependecies into account. This still works as expected, also the build output is as expected.

Secondly, it builds the apps in parallel. Since the upgrade, this now fails at random times, throwing random error messages (these are 3 examples from separate runs):

NX Successfully ran target build for project authentication and 12 tasks it depends on
Nx read the output from the cache instead of running the command for 4 out of 13 tasks.
NX ENOENT: no such file or directory, copyfile '/home/jenkins/workspace/Web/CubeMobileApps@6/Webpages/CubeMobileApps/node_modules/.cache/nx/2a68e5917b96f4d3a9faba11b7d7df32d791dbe6ee9406d47f4c319d7c2bc892/outputs/dist/libs/inputs/esm2020/lib/components/dma-input-textarea/index.mjs' -> '/home/jenkins/workspace/Web/CubeMobileApps@6/Webpages/CubeMobileApps/dist/libs/inputs/esm2020/lib/components/dma-input-textarea/index.mjs'

NX Successfully ran target build for project jobs and 12 tasks it depends on
Nx read the output from the cache instead of running the command for 6 out of 13 tasks.
NX ENOTEMPTY: directory not empty, rmdir '/home/jenkins/workspace/Web/CubeMobileApps/Webpages/CubeMobileApps/dist/libs/ui/styles/scss'

NX Successfully ran target build for project jobs and 12 tasks it depends on
Nx read the output from the cache instead of running the command for 10 out of 13 tasks.
NX ENOENT: no such file or directory, chmod '/home/jenkins/workspace/Web/CubeMobileApps@3/Webpages/CubeMobileApps/dist/libs/common-components/esm2020/lib/table/index.mjs'

It crashes on dependent libs, although (and I verified it) these libs have been build in the first step, so they should be in the Nx cache. I think it rebuilds them for some reason, which causes now issues when building the apps in parallel, because multiple apps depend on the same libs.

Expected Behavior

After building all publishable libs, building apps in parallel shouldn't give issues (used to work in Nx 14 and older).

GitHub Repo

No response

Steps to Reproduce

1. Have an Nx monorepo 15.9.2 with publishable Angular libs and Angular apps
2. Build all the Angular libs sequentially taking into account the order of dependencies
3. Build all the Angular apps simultaneously in parallel (we throttle on max 5)

Nx Report

>  NX   Report complete - copy this into the issue template

   Node : 16.16.0
   OS   : win32 x64
   npm  : 8.1.3

   nx                      : 15.9.2
   @nrwl/js                : 15.9.2
   @nrwl/jest              : 15.9.2
   @nrwl/linter            : 15.9.2
   @nrwl/workspace         : 15.9.2
   @nrwl/angular           : 15.9.2
   @nrwl/cli               : 15.9.2
   @nrwl/cypress           : 15.9.2
   @nrwl/devkit            : 15.9.2
   @nrwl/eslint-plugin-nx  : 15.9.2
   @nrwl/tao               : 15.9.2
   @nrwl/webpack           : 15.9.2
   typescript              : 4.9.5
   ---------------------------------------
   Community plugins:
   @nguniversal/express-engine : 15.2.0

Note that our CI pipeline is on Linux

Failure Logs

No response

Additional Information

No response

[yinshuxun]

@wimme I have also encountered a similar problem, have you found a solution to the problem?

[github-actions]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs.
If we missed this issue please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.