-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Buffered serialization #2225
Buffered serialization #2225
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW: We should also refactor parquetWrtier to use the file_writer.
18cd20c
to
f2d05f4
Compare
Can that be left to a later PR, or do you want it included in this one? |
f2d05f4
to
e73855d
Compare
I think i can do it in later PRs. |
e73855d
to
ab45436
Compare
ab45436
to
32829cb
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #2225 +/- ##
==========================================
+ Coverage 89.67% 89.69% +0.01%
==========================================
Files 1008 1012 +4
Lines 35830 35862 +32
==========================================
+ Hits 32131 32166 +35
+ Misses 3699 3696 -3
☔ View full report in Codecov by Sentry. |
Splits the SerDeser class into a Serializer and Deserializer, and a BufferedFileWriter (for serializing to disk), and BufferedSerializer (for serializing to an in-memory buffer).