-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HDFS-17573. Allow turn on both FSImage parallelization and compression #6929
Conversation
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
it seems two unit tests ( |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
16099a1
to
31fa957
Compare
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
@sodonnel @Hexiaoqiao Hello, may i request review on this? |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Leave some nit comments inline. PFYI.
.../hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
Show resolved
Hide resolved
.../java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshotParallelAndCompress.java
Outdated
Show resolved
Hide resolved
.../java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshotParallelAndCompress.java
Outdated
Show resolved
Hide resolved
.../java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshotParallelAndCompress.java
Outdated
Show resolved
Hide resolved
...adoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
Outdated
Show resolved
Hide resolved
🎊 +1 overall
This message was automatically generated. |
@Hexiaoqiao @sodonnel |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. +1 from my side. Will try to checkin if no more other comments while waiting two work days.
@Hexiaoqiao Thanks a lot for reviewing! |
Committed to trunk. Thanks @Last-remote11 for your works! |
Description of PR
The feature added HDFS-14617(in Improve FSImage load time by writing sub-sections to the FSImage index. by Stephen O'Donnell) makes loading FSImage very faster.
But this option cannot be activated when turn on dfs.image.compress=true.
In my opinion, larger clusters require both settings at the same time.
In my environment, the cluster I'm using has approximately 6 million file system objects and FSImage is approximately 11GB with dfs.image.compress=true setting. If turn off the dfs.image.compress option, it is expected to exceed 30GB, in which case it will take a long time to move FSImage from standby to active namenode using high network resource.
It was proved in this jira(HDFS-16147 by kinit) that loading FSImage parallel and FSImage compression can be turned on at the same time. (And worked well on my environment also.)
I created this new Jira and PR because the discussion in HDFS-16147 ended in 2021, and I want it to be officially added in the next release, instead of patch available.
The actual code of the patch was written by kinit and I resolved empty sub-section problem(see below comment of HDFS-16147) and added test code.
If this is not a proper method, please let me know another way to contribute.
Thanks.
How was this patch tested?
added
testParallelSaveAndLoadWithCompression
and ran unit test.For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?