Namenode and Journalnodes are not restarting

[#1]

Hello 

Can some one please help me on this issue,

My two journal nodes are not in sync and i cant able to restart them and both the namenodes are down.

I can see below logs in NN,

FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog
Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.10.10.10:8485, 10.10.10.11:8485, 10.10.10.09:8485], stream=null))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:
10.10.10.09:8485: lastPromisedEpoch: 69
httpPort: 8480 fromURL: "http://Master_Host(10.10.10.09):8480"

When i use to do restart of JN and not happening,

And below are the Journalnode logs,

ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode
RECEIVED SIGNAL 15: SIGTERM
WARN org.apache.hadoop.metrics2.impl.MetricsConfig
Cannot locate configuration: tried hadoop-metrics2-journalnode.properties,hadoop-metrics2.properties

Can some one please suggest me.

Thanks,

Vinod

[#2]

Hello,

Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.10.10.10:8485, 10.10.10.11:8485, 10.10.10.09:8485], stream=null))

As you can see above that instead of FQDN, the IP shows up, which means that when the JN is trying to talk to DNS to resolve the IP to the FQDN, it failed. It looks like your DNS have issues resolving the IPs. Please look into it.

[#3 · Accepted solution]

Hello thanks for your responce, Actually the issue is resolved and its JournalNodeEdits directory permission issue.

So i have modified it and restarted Jornalnodes and started successfully.

NN's are also came back and able to see all other dependent services.

Thanks again 🙂