Created on 12-06-201601:02 PM - edited 08-17-201907:41 AM
SYMPTOM: While performing step for Namenode HA, on the step of 'hdfs namenode -initializeSharedEdits' it failed with below error -
ERROR:
[root@localhost conf]# sudo su hdfs -l -c 'hdfs namenode -initializeSharedEdits'
16/11/22 09:43:56 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = locahost/10.200.206.12
STARTUP_MSG: args = [-initializeSharedEdits]
STARTUP_MSG: version = 2.7.1.2.3.0.0-2557
STARTUP_MSG: classpath = /usr/hdp/2.3.0.0-2557/hadoop/conf:/usr/hdp/2.3.0.0-2557/hadoop/lib/commons-cli-1.2.jar:/us r/hdp/2.3.0.0-2557/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.0.0-2557/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.3.0.0-2557/
.
.
.
STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 9f17d40a0f2046d217b2bff90ad6e2fc7e41f5e1; compiled by 'jenkins' on 2015-07-14T13:08Z
STARTUP_MSG: java = 1.8.0_51
************************************************************/
16/11/22 09:43:56 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/11/22 09:43:56 INFO namenode.NameNode: createNameNode [-initializeSharedEdits]
16/11/22 09:43:56 ERROR namenode.NameNode: No shared edits directory configured for namespace null namenode null
16/11/22 09:43:56 INFO util.ExitUtil: Exiting with status 0
16/11/22 09:43:56 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/10.200.206.12
************************************************************/
ROOT CAUSE: While performing NN Ha we found that Step 5 "Configure Components" is getting executed in a second, which was issue suspected. It was not stopping services and performing desired steps mentioned in screenshot below -
RESOLUTION: Suspected that Ambari server was holding cache in DB. Clearing the Ambari server cache using below steps resolved the issue -
# log out from the ambari server
# list persisted k-v(ui cache)
$curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist
# get the cached state:
$curl -i -u admin:admin -H 'X-Requested-By: ambari' -X GET http://localhost:8080/api/v1/persist/CLUSTER_CURRENT_STATUS
# reset/clean the cache
$curl -u admin:admin -H 'X-Requested-By:ambari' -d '{"CLUSTER_CURRENT_STATUS": "{\"clusterState\":\"DEFAULT\"}" }' -X POST 'http://localhost:8080/api/v1/persist'