HDFS-16690. Automatically format unformatted JNs with JournalNodeSyncer #6925
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
aswinmprabhu
force-pushed
the
aswinmprabhu/HDFS-16690
branch
from
30257ae
to
809e51a
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
reviewed
Jul 16, 2024
There was a problem hiding this comment.
Thanks @aswinmprabhu for your proposal. Good idea. Leave some comments inline. PFYI.
| journal.getStorage().getEditsSyncDir()); | ||
| return; | ||
| } | ||
| continue; |
There was a problem hiding this comment.
If it will drop into endless loop when tryFormatting set to false by default here?
There was a problem hiding this comment.
The endless loop already exists. If the JN is unformatted, it will throw an exception in a loop every time the syncer tries to sync from other JNs. tryFormatting doesn't add anything extra to the loop. But if it is set, the formatting could happen.
| return GetEditLogManifestResponseProto.newBuilder() | ||
| .setManifest(PBHelper.convert(manifest)) | ||
| .setHttpPort(jn.getBoundHttpAddress().getPort()) | ||
| .setFromURL(jn.getHttpServerURI()) | ||
| .build(); | ||
| } | ||
|
|
||
| @Override | ||
| public StorageInfoProto getStorageInfo(String jid, | ||
| String nameServiceId) throws IOException { |
There was a problem hiding this comment.
nameServiceId here is not used anymore, so why we should define it here?
There was a problem hiding this comment.
Good catch. This was a miss. getOrCreateJournal should be called with jid and nameServiceId. I've pushed an update. Thanks!
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@Hexiaoqiao can you please take a look again? I've replied and updated the PR. |
|
💔 -1 overall
This message was automatically generated. |
Hexiaoqiao
approved these changes
Jul 20, 2024
There was a problem hiding this comment.
LGTM. +1. The failed unit test is not related to this PR (which is traced by #6951). Will commit if no more other comments for two work days.
|
Thanks, @Hexiaoqiao for the approval. Can we merge at the end of today? |
|
Committed to trunk. Thanks @aswinmprabhu for your works. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
If an unformatted JournalNode is added to an existing JournalNode set --
This scenario can arise in different situations like
Manually fixing this involves rsyncing the VERSION file to the edit log root directory from the other healthy JNs. A similar issue concerning the paxos directory was solved in HDFS-10659.
This PR tries to leverage the already existing JournalNodeSyncer daemon to format the JournalNode on which it is running when it discovers that syncs can't happen due to the
JournalNotFormattedException. JournalNodeSyncer calls theformatWithSyncermethod if it sees that the JN is unformatted.formatWithSyncerwill loop over the other JN proxies, trying to fetch theStorageInfoobject from them. The StorageInfo object is then used to format the JN by callingJNStorage.format().How was this patch tested?
Unit tests have been added for the change:
Build is also successful:
I've tested the changes manually in a K8S cluster with 3 JNs:
JN root dir before testing:
Logs show that JN can receive edit logs:
2024-07-04 08:50:55,242 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /grid/edits/journal/d ata/asprabhu-hadoop/current/edits_inprogress_0000000000000000001 -> /grid/edits/journal/data/asprabhu-hadoop/current/edits_00000000 00000000001-0000000000000000042Killed the JN process and deleted the edit logs root dir:
Started the JN process. The syncer formatted the JN. Some relevant log lines:
Syncer was also able to fill the holes from other JNs:
JN root dir after the testing:
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?