You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The troubleshooting for decommission is very hard even in DEBUG mode.
In somehow cases when decommission has taken a lot of time but we are not sure it is still in-progress, we run -refreshNodes to try to re-trigger the decommission(In fact it is useless).
Then we check the NameNode logs, but it is a pity that we cannot find any useful log about whether or not our refreshNodes subcommand has taken any effect or not.
So, my change is:
I changed this critical log 's level from TRACE to INFO since this log is not a repeatedly log and it gives administrator critical information for about what happened.
When the Reconstruction which is trigged by node decommission is skipped, we want to know the reason. The reason is divided to 3 catagories
No source node is available
No Target node is available
ReconstructionWork is built but validation failed
I put these reasons in a single enum ReconstructionSkipReason. In DEBUG mode, the reason will be aggregated and showed to users.
The log is as below:
2024-07-10 02:59:09,707 [Thread-990] DEBUG blockmanagement.BlockManager: Block blk_3_0 is not scheduled for reconstruction since: [
source node or storage unavailable on node [DISK]storageID_0_3:NORMAL:127.0.0.1:9866. Detail : [stored replica state is corrupt or excess]
source node or storage unavailable on node [DISK]storageID_1_3:NORMAL:127.0.0.1:9866. Detail : [replica is already decommissioned]
]
VicoWu
changed the title
Change repeated decommission log level to info
Change Ineffective Repeated Decommission Operation Log Level to INFO
Jul 4, 2024
VicoWu
changed the title
Change Ineffective Repeated Decommission Operation Log Level to INFO
Change Uneffective Repeated Decommission Operation Log Level to INFO
Jul 4, 2024
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚
mvninstall
50m 1s
trunk passed
+1 💚
compile
1m 25s
trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
compile
1m 18s
trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
checkstyle
1m 13s
trunk passed
+1 💚
mvnsite
1m 26s
trunk passed
+1 💚
javadoc
1m 12s
trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javadoc
1m 48s
trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
spotbugs
3m 32s
trunk passed
+1 💚
shadedclient
51m 15s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚
mvninstall
1m 43s
the patch passed
+1 💚
compile
1m 49s
the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javac
1m 49s
the patch passed
+1 💚
compile
1m 42s
the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
javac
1m 42s
the patch passed
+1 💚
blanks
0m 0s
The patch has no blanks issues.
+1 💚
checkstyle
1m 35s
the patch passed
+1 💚
mvnsite
1m 28s
the patch passed
+1 💚
javadoc
0m 59s
the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javadoc
1m 44s
the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
VicoWu
changed the title
Change Uneffective Repeated Decommission Operation Log Level to INFO
HDFS-17568 Change Uneffective Repeated Decommission Operation Log Level to INFO
Jul 4, 2024
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚
mvninstall
49m 49s
trunk passed
+1 💚
compile
1m 25s
trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
compile
1m 14s
trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
checkstyle
1m 12s
trunk passed
+1 💚
mvnsite
1m 22s
trunk passed
+1 💚
javadoc
1m 9s
trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javadoc
1m 41s
trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
spotbugs
3m 18s
trunk passed
+1 💚
shadedclient
41m 38s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚
mvninstall
1m 10s
the patch passed
+1 💚
compile
1m 18s
the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javac
1m 18s
the patch passed
+1 💚
compile
1m 8s
the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
javac
1m 8s
the patch passed
+1 💚
blanks
0m 0s
The patch has no blanks issues.
+1 💚
checkstyle
1m 3s
the patch passed
+1 💚
mvnsite
1m 12s
the patch passed
+1 💚
javadoc
0m 56s
the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚
javadoc
1m 37s
the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚
spotbugs
3m 20s
the patch passed
+1 💚
shadedclient
42m 3s
patch has no errors when building and testing our client artifacts.
VicoWu
changed the title
HDFS-17568 Change Uneffective Repeated Decommission Operation Log Level to INFO
HDFS-17568 Refined Log for Decommissioning Nodes and Reason for Not Finding Src Node
Jul 6, 2024
VicoWu
changed the title
HDFS-17568 Refined Log for Decommissioning Nodes and Reason for Not Finding Src Node
HDFS-17568 [Decommission]Show Aggregated Reason for Why Low-Redundancy Block is Skipped for Reconstruction
Jul 10, 2024
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The troubleshooting for decommission is very hard even in DEBUG mode.
In somehow cases when decommission has taken a lot of time but we are not sure it is still in-progress, we run
-refreshNodes
to try to re-trigger the decommission(In fact it is useless).Then we check the NameNode logs, but it is a pity that we cannot find any useful log about whether or not our
refreshNodes
subcommand has taken any effect or not.So, my change is:
I put these reasons in a single enum
ReconstructionSkipReason
. In DEBUG mode, the reason will be aggregated and showed to users.The log is as below: