Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: make sure to write entire buffer #49211

Closed
wants to merge 0 commits into from

Conversation

atlowChemi
Copy link
Member

fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

Refs: #42434 (comment)

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Aug 17, 2023
lib/internal/fs/streams.js Outdated Show resolved Hide resolved
lib/internal/fs/streams.js Outdated Show resolved Hide resolved
@mscdex mscdex added the needs-benchmark-ci PR that need a benchmark CI run. label Aug 17, 2023
@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 20, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 20, 2023
@nodejs-github-bot
Copy link
Collaborator

@atlowChemi
Copy link
Member Author

benchmark output:

                                                                                              confidence improvement accuracy (*)    (**)   (***)
fs/write-stream-throughput.js size=1024 encodingType='asc' dur=5                                             -0.97 %       ±2.40%  ±3.19%  ±4.15%
fs/write-stream-throughput.js size=1024 encodingType='buf' dur=5                                              0.81 %       ±1.53%  ±2.03%  ±2.65%
fs/write-stream-throughput.js size=1024 encodingType='utf' dur=5                                              1.61 %       ±1.75%  ±2.33%  ±3.06%
fs/write-stream-throughput.js size=1048576 encodingType='asc' dur=5                                           0.09 %       ±2.32%  ±3.09%  ±4.02%
fs/write-stream-throughput.js size=1048576 encodingType='buf' dur=5                                          -0.38 %       ±0.60%  ±0.80%  ±1.03%
fs/write-stream-throughput.js size=1048576 encodingType='utf' dur=5                                          -0.38 %       ±1.50%  ±1.99%  ±2.60%
fs/write-stream-throughput.js size=2 encodingType='asc' dur=5                                                -0.31 %       ±2.03%  ±2.71%  ±3.53%
fs/write-stream-throughput.js size=2 encodingType='buf' dur=5                                                -0.53 %       ±1.74%  ±2.32%  ±3.02%
fs/write-stream-throughput.js size=2 encodingType='utf' dur=5                                                -0.13 %       ±2.22%  ±2.96%  ±3.85%
fs/write-stream-throughput.js size=65535 encodingType='asc' dur=5                                     **     -1.64 %       ±1.05%  ±1.40%  ±1.85%
fs/write-stream-throughput.js size=65535 encodingType='buf' dur=5                                             0.09 %       ±1.02%  ±1.37%  ±1.80%
fs/write-stream-throughput.js size=65535 encodingType='utf' dur=5                                             0.06 %       ±1.18%  ±1.57%  ±2.04%


Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 142 comparisons, you can thus
expect the following amount of false-positive results:
  7.10 false positives, when considering a   5% risk acceptance (*, **, ***),
  1.42 false positives, when considering a   1% risk acceptance (**, ***),
  0.14 false positives, when considering a 0.1% risk acceptance (***)

Copy link
Member

@ronag ronag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing tests pass and it's quite tricky make tests for this. You probably need to pass a custom "fs" implementation into the stream which will return EAGAIN.

@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@nodejs-github-bot
Copy link
Collaborator

@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

lib/internal/fs/streams.js Outdated Show resolved Hide resolved
@atlowChemi atlowChemi force-pushed the write-all branch 2 times, most recently from 2c9b7e0 to 20d51d2 Compare August 23, 2023 13:44
@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 23, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 23, 2023
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@atlowChemi atlowChemi added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Aug 24, 2023
atlowChemi added a commit to atlowChemi/node that referenced this pull request Aug 24, 2023
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: nodejs#49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
@atlowChemi
Copy link
Member Author

Landed in feb5b0f

@atlowChemi atlowChemi closed this Aug 24, 2023
@atlowChemi atlowChemi deleted the write-all branch August 24, 2023 05:36
UlisesGascon pushed a commit that referenced this pull request Sep 10, 2023
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: #49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
@UlisesGascon UlisesGascon mentioned this pull request Sep 10, 2023
richardlau pushed a commit that referenced this pull request Sep 28, 2023
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: #49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
This was referenced Oct 6, 2023
alexfernandez pushed a commit to alexfernandez/node that referenced this pull request Nov 1, 2023
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: nodejs#49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. fs Issues and PRs related to the fs subsystem / file system. needs-benchmark-ci PR that need a benchmark CI run. needs-ci PRs that need a full CI run.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants