🐛 BUG: Windows+node:fs=> ALL Pages uploads/updates fail with 2000+ files [ERROR EMFILE]

[torcd]

System tested

OS: Windows 64-bit
pm: nvm-windows
node.js: 16.7.0 (lowest supported), 16.16.0 (lts), 17.9.0, 18.5.0, 18.7.0 (latest), likely all versions affected
wrangler: 2.0.17, 2.0.23, likely all versions >2.0.6 affected
dependencies: all updated to "wanted" versions
ERROR src: -node:fs, -chokidar, -wrangler pages publish/upload code (eg lack of graceful fs queuing/retries)

Issue:

On win64, when publishing a few thousand files via wrangler pages publish, the uploads fail due to chronic ERROR EMFILE: too many files open '<C:\path\to\last\read\file>' errors.

The uploads either fail:

1. during the dir walk/read/check stage if a previous deployment exceeds [6000-8000] files -> thus, one can't reliably update even a single file if larger previous deployments went through

OR

2. during the upload stage when the total batch of files is in the [2K-4K]+ range -> ie, even the first deployment will fail if you start with a larger batch

Similar bugs/fixes elsewhere:

#141 chokidar: Use graceful-fs to avoid EMFILE?
#624 IPFS: upload directory with subdirectories [WIP]

Potential Fix/Solution:

The bug likely stems from use of the core node fs module and how files are watched and read in the pipeline. While chokidar+fsevents might be preventing EMFILE errors on Mac, no such mitigations appear to be used on Windows systems. Instead of queuing/waiting/retrying, the uploads fail entirely once a single EMFILE error is thrown, rendering Pages projects with a few thousand files impossible to maintain even though wrangler technically allows 20,000 files per upload. :((

Can a test build with graceful-fs +wrangler publish/upload code fs replacements be temporarily made available at least until a permanent Windows-specific fix is released? Thanks.

[Cherry]

Ran into this myself today with a large project. graceful-fs seems to be a good drop-in replacement to queue up the calls and more gracefully handle EMFILE errors.