-
-
Notifications
You must be signed in to change notification settings - Fork 597
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stringify stats using streaming approach #2190
Stringify stats using streaming approach #2190
Conversation
I think node 12+ support is too optimistic until April. |
@TrySound Didn't know that requirement. Will fix it a bit later. |
I've re-run the CI job, |
Change the approach for stats stringify to use streams instead of JSON.stringify() + sync write. No API or functionality is changed. It makes stats available for huge projects. Changes has the follow effects: Memory Allows to avoid max string length limit of V8 engine (500MB). In other words, stats becomes available for huge projects, whose stats exceed 500MB. Reduces memory consumption on serialization. Adds time penalty. However, this make sense for huge stats only and seems to be an acceptable, given that otherwise stats may be unavailable (due max string limit hit or out of memory). Adds new dependency "json-ext", which is dependency free and most effective solution for a huge JSON parse/stringify at the moment, and other solutions have troubles to handle huge JSON (see https://github.com/discoveryjs/json-ext/blob/master/benchmarks/README.md#stream-stringifying).
Codecov Report
@@ Coverage Diff @@
## master #2190 +/- ##
==========================================
+ Coverage 66.09% 68.32% +2.23%
==========================================
Files 72 72
Lines 2345 2346 +1
Branches 518 518
==========================================
+ Hits 1550 1603 +53
+ Misses 795 743 -52
Continue to review full report at Codecov.
|
Fixed, now json-ext supports Node.js 10. All tests passed |
Big thanks |
What kind of change does this PR introduce?
This PR changes the approach for stats stringify to use stream instead of JSON.stringify() + sync write. No API or functionality is changed.
Did you add tests for your changes?
No
If relevant, did you update the documentation?
Not needed
Summary
It makes stats available for huge projects. Changes has the follow effects:
Does this PR introduce a breaking change?
No
Other information
Tested on AST Explorer (https://github.com/fkling/astexplorer)
Command (cwd website):
BEFORE:
Time: 4583ms
Memory used: +1_850_027_831 (heap: +1_508_459_432, external: +341_568_399)
AFTER:
Time: 5881ms
Memory used: -90_171_587 (heap: -32_339_984, external: -57_831_603)
So it's a bit slower, but much effective in memory consumption.