Reply
Highlighted
Explorer
Posts: 24
Registered: ‎01-11-2018

Enabling HDFS HA fails because of empty journal id (CDH 5.9.3)

[ Edited ]

Hi! I'm trying to configure HDFS HA on CDH 5.9.3 cluster with help of Cloudera Manager, but it failes on stage 'Initializing shared edits directory of NameNodes.'

Failed to initialize Shared Edits Directory of NameNode NameNode (node-1). Initialization can fail if the Shared Edits Directory is not empty. Check the stderr log for details.:
java.lang.IllegalArgumentException: Unable to construct journal, qjournal://XXX.XXX.XXX.XXX:8485;YYY.YYY.YYY.YYY:8485;ZZZ.ZZZ.ZZZ.ZZZ:8485/
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1634)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:283)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:261)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:800)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:639)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:291)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1097)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1632)
    ... 13 more
Caused by: java.lang.IllegalArgumentException: bad journal id:
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.checkJournalId(QuorumJournalManager.java:164)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.parseJournalId(QuorumJournalManager.java:159)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:366)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:150)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:117)
    at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:106)



Referring to https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_hag_hdfs_ha_enabling.html the Journal ID should be present at the end of qjournal identifier, i.e. qjournal://XXX.XXX.XXX.XXX:8485;YYY.YYY.YYY.YYY:8485;ZZZ.ZZZ.ZZZ.ZZZ:8485/nameservice1 but appears to be empty. When I look at HDFS configuration, both 'NameNode Nameservice' and 'SecondaryNameNode Nameserice' are set to 'nameservice1', and I set this name at the first step of HA configuration. Has anybody encountered problem like this in the past? Is it a problec caused by Cloudera Manager or our HDFS configuration? Any help is very appreciated :)

Announcements