question Namenode and Journalnodes are not restarting in Support Questions

Namenode and Journalnodes are not restarting

kvinod — Mon, 11 Feb 2019 07:27:39 GMT

Hello

Can some one please help me on this issue,

My two journal nodes are not in sync and i cant able to restart them and both the namenodes are down.

I can see below logs in NN,

FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog
Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.10.10.10:8485, 10.10.10.11:8485, 10.10.10.09:8485], stream=null))
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:
10.10.10.09:8485: lastPromisedEpoch: 69
httpPort: 8480 fromURL: "http://Master_Host(10.10.10.09):8480"

When i use to do restart of JN and not happening,

And below are the Journalnode logs,

ERROR org.apache.hadoop.hdfs.qjournal.server.JournalNode
RECEIVED SIGNAL 15: SIGTERM
WARN org.apache.hadoop.metrics2.impl.MetricsConfig
Cannot locate configuration: tried hadoop-metrics2-journalnode.properties,hadoop-metrics2.properties

Can some one please suggest me.

Thanks,

Vinod

Re: Namenode and Journalnodes are not restarting

TonyStank — Mon, 11 Feb 2019 13:10:50 GMT

Hello,

Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.10.10.10:8485, 10.10.10.11:8485, 10.10.10.09:8485], stream=null))

As you can see above that instead of FQDN, the IP shows up, which means that when the JN is trying to talk to DNS to resolve the IP to the FQDN, it failed. It looks like your DNS have issues resolving the IPs. Please look into it.

Re: Namenode and Journalnodes are not restarting

kvinod — Tue, 12 Feb 2019 14:56:25 GMT

Hello thanks for your responce, Actually the issue is resolved and its JournalNodeEdits directory permission issue.

So i have modified it and restarted Jornalnodes and started successfully.

NN's are also came back and able to see all other dependent services.

Thanks again 🙂