question ZooKeeper Failover controller crashes when the Hadoop NameNode goes down in Support Questions

question ZooKeeper Failover controller crashes when the Hadoop NameNode goes down in Support Questions https://community.cloudera.com/t5/Support-Questions/ZooKeeper-Failover-controller-crashes-when-the-Hadoop/m-p/213261#M175189 We are setting up a Hadoop Cluster in our development environment. While we were testing the fail over of the NameNode we noticed that the Zookeeper Failover controller would sometimes crash. In one case both ZooKeeper and the ZooKeeper Failover controller crashed.At one point in our testing both NameNodes were in an active state. This would cause a split brain scenario in the Hadoop Cluster. We have not seen any useful information in the logs. We are using the following versions: - hadoop-2.7.3 - zookeeper-3.4.10We have two a four server cluster. Two of the servers are dedicated to NameNode and two of the servers are dedicated to DataNodes.The components running on the NameNode servers are - NameNode - ZooKeeper - ZooKeeper Failover controller - JournalNodeThe components running on the DataNode servers are - DataNode - ZooKeeper - JournalNodeThe following matrix contains the test scenarios. After the matrix we have the contents of the core-site.xml and hdfs-site.xml. <TABLE> <TBODY><TR> <TD>NameNode Server 1 </TD> <TD>NameNode Server 2</TD> </TR> <TR> <TD>Name node 1 is active </TD> <TD>NameNode 2 is standby</TD> </TR> <TR> <TD>Kill -9 pid NameNode 1 Name Node 1 is down</TD> <TD>NameNode 2 is active</TD> </TR> <TR> <TD>Start NameNode 1 Zoo Keeper Failover 1 crashes NameNode 1 is standby</TD> <TD>NameNode 2 is active</TD> </TR> <TR> <TD>Start Zoo Keeper Failover Controller1 NameNode 1 is active</TD> <TD>NameNode 2 is standby</TD> </TR> <TR> <TD>Kill -9 pid NameNode 1 Name Node 1 is down</TD> <TD>NameNode 2 is active</TD> </TR> <TR> <TD>Start NameNode 1 Zoo Keeper Failover 1 crashes NameNode 1 is standby No useful information in the Zoo Keeper Failover Controller Logs</TD> <TD>NameNode 2 is Aactive</TD> </TR> <TR> <TD>Turn on Log4j debugging Start Zoo Keeper Failover Controller1 Zoo Keeper Fail Over Controller does not start </TD><TD>NameNode 2 is Aactive</TD> </TR> <TR> <TD>Turn on Log4j debugging,console Start Zoo Keeper Failover Controller1 Zoo Keeper Fail Over Controller does not start Logs: unable to start failover controller Parent znode does not exist Logs: run with -formatZK to initalize Zookeeper</TD> <TD>NameNode 2 is active</TD> </TR> <TR> <TD>run: hdfs zkfc -formatZK Start Zoo Keeper Failover Controller 1 Name node 1 is active </TD> <TD>NameNode 2 is active </TD></TR> <TR> <TD>Name node 1 is active </TD> <TD>Stop Name Node 2 Zoo Keeper Failover Controller 2 crashed Zoo Keeper crashed</TD> </TR> <TR> <TD>NameNode 1 is standby</TD> <TD>Start Zoo Keeper Start Zoo Keeper Failover Controller 2 Start Name Node 2 Name Node 2 active</TD></TR></TBODY></TABLE>core-site.xml<PRE><configuration> <property> <name>hadoop.tmp.dir</name> <value>/opt/xpm/hadoop_tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://ha-cluster</value> </property> <property> <name>dfs.jornalnode.edits.dir</name> <value>/opt/xpm/hadoop_journal</value> </property> </configuration> </PRE>hdfs-site.xml<PRE><configuration> <property> <name>dfs.namenode.name.dir</name> <value>/opt/xpm/hadoop_namenode</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.nameservices</name> <value>ha-cluster</value> </property> <property> <name>dfs.ha.namenodes.ha-cluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.ha-cluster.nn1</name> <value>r00j9rn0c.bnymellon.net:9000</value> </property> <property> <name>dfs.namenode.rpc-address.ha-cluster.nn2</name> <value>r00j9sn0c.bnymellon.net:9000</value> </property> <property> <name>dfs.namenode.http-address.ha-cluster.nn1</name> <value>r00j9rn0c.bnymellon.net:50070</value> </property> <property> <name>dfs.namenode.http-address.ha-cluster.nn2</name> <value>r00j9sn0c.bnymellon.net:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://r00j9rn0c.bnymellon.net:8485;r00j9sn0c.bnymellon.net;r00j9tn0c.bnymellon.net:8485;r00j9un0c.bnymellon.net:8485/ha-cluster</value> </property> <property> <name>dfs.client.failover.proxy.provider.ha-cluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>r00j9rn0c.bnymellon.net:2181,r00j9sn0c.bnymellon.net:2181,r00j9tn0c.bnymellon.net:2181,r00j9un0c.bnymellon.net:2181</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/users/home/pkimd1m/.ssh/id_rsa</value> </property> </configuration> </PRE> Tue, 08 Aug 2017 03:02:09 GMT albert_stark 2017-08-08T03:02:09Z