Cluster bricks when all journalnodes are down

[nightkr]

Affected version

main

Current and expected behavior

The JournalNodes seem to get stuck in a crashloop if all are deleted at the same time, complaining about not being able to find missing edits.

Possible solution

No response

Additional context

No response

Environment

No response

Would you like to work on fixing this bug?

None

[sbernauer]

Good news:
IIRC this happened when demo-ing Kerberos support, I did also run into this problem.
The current WIP state of hdfs Kerberos has solved the problem and added kuttl tests to test this scenario. Linking the PR als fix

[sbernauer]

The problem was the journalnodes rejecting the namenodes because of reverse DNS roulette

org.apache.hadoop.hdfs.server.common.HttpGetFailedException: Fetch of https://hdfs-journalnode-default-0.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true failed with status code 403
Response message:
Only Namenode and another JournalNode may access this servlet

resulting in

2023-05-30 09:23:01,651 ERROR namenode.RedundantEditLogInputStream (RedundantEditLogInputStream.java:nextOp(233)) - Got error reading edit log input stream https://hdfs-journalnode-default-0.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true; failing over to edit log https://hdfs-journalnode-default-1.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true

The got premature end-of-file at txid 0; expected file to go up to 10 was because of the failover

I solved the problem by not using _HOST in the principals, so that journalnode did not reject the namenode

[sbernauer]

Closing this, as we fixed the problem in the Kerberos feature-branch. Please feel free to re-open when the problem reappears!