Speedy

[organizedgrime]

In this branch we diagnosed and addressed issues regarding performance. GZIP is now replaced by ZSTD, and duplication detection and file grouping are now outsources to fclones. As a result, we've drastically improved performance and are now packing files at ~1GiB/s.

[codecov]

Codecov Report

Patch coverage: 2.94% and project coverage change: +0.53 🎉

Comparison is base (02f3692) 5.06% compared to head (07550bf) 5.60%.

❗ Current head 07550bf differs from pull request most recent head eb3ed06. Consider uploading reports for the commit eb3ed06 to get more accurate results

Additional details and impacted files

@@            Coverage Diff            @@
##           master    #133      +/-   ##
=========================================
+ Coverage    5.06%   5.60%   +0.53%     
=========================================
  Files          15      13       -2     
  Lines         454     375      -79     
  Branches      454     375      -79     
=========================================
- Hits           23      21       -2     
+ Misses        431     354      -77     

Impacted Files	Coverage Δ
...rc/do_pipeline_and_write_metadata/pack_pipeline.rs	0.00% <0.00%> (ø)
.../do_pipeline_and_write_metadata/unpack_pipeline.rs	0.00% <0.00%> (ø)
dataprep-lib/src/lib.rs	0.00% <0.00%> (ø)
dataprep-lib/src/spider.rs	0.00% <0.00%> (ø)
dataprep-lib/src/types/pack_plan.rs	0.00% <0.00%> (ø)
dataprep-lib/src/types/shared.rs	0.00% <0.00%> (ø)
dataprep-lib/src/types/spider.rs	0.00% <0.00%> (ø)
dataprep-lib/src/types/unpack_plan.rs	0.00% <0.00%> (ø)
dataprep-lib/src/utils/fs.rs	0.00% <ø> (ø)
dataprep-lib/src/vacuum/pack.rs	0.00% <0.00%> (ø)
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.