unable to start name node

[#1]

Hi Team,

When we initiate the hadoop cluster services using Ambari after enabling kerborose getting authentication failed, logs have attached.

Please help us on this.

[#2]

@hadoop hdfs

Did you install JCE to unlimited extension?

Have you tried restarting the cluster?

If it still fails can you run the below command

$ su - hdfs 
$ hadoop namenode -recover 

Please revert

[#3]

Thanks for the reply Geoffrey Shelton,

Did you install JCE to unlimited extension?

--> We have installed jce_policy-8.zip

#unzip -o -j -q <</fullpath>>/jce_policy-8.zip -d /usr/java/jdk1.8.0/jre/lib/security/

Have you tried restarting the cluster?

--Yes

Our hadoop cluster services were running fine without any issues, after enabling kerborose, we are not able to start the services.

When we install kerborose at the last step it will try to restart all the cluster services and some of the services will not come up and operation will fail.

[#4]

Hi Geoffrey,

We have issued namenode recover command, now all the services are up and running fine ( with kerborose enabled).

We have 3 data nodes and all the data nodes are up, now we are getting below alert on datanode1.

Connection failed to http://rjio-prodcluster-datanode01:1022 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -c /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -w '%{http_code}' http://rjio-prodcluster-datanode01:1022 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpcLoFod 2>/tmp/tmp7T5zRl' returned 7.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to rjio-prodcluster-datanode01:1022; Connection refused

[#5]

Latest log from datanode1:

Connection failed: [Errno 111] Connection refused to rjio-prodcluster-datanode01:1019

[#6]

@hadoop hdfs

Any updates?

[#7]

error-log1.pngerror-log2.png@Geoffrey,

We have started the all the services successfully. datanode1 services are started but in ambari UI and hdfs admin report its not showing (showing it as dead node).

We have Two Namenodes & Three data nodes.

Error log and report log have been attached.

report-3.txt

[#8]

@hadoop hdfs
Can you run the below command and then the report to see if the internal view was refreshed. Note dfsadmin are admin scripts so run with caution

$ hdfs dfsadmin -refreshNodes

Please let me know !

[#9]

@Geoffrey shelton Okot,

After refreshing the nodes we have kill all the processes and started from Ambari, Now all the services are up & running fine.

Thank you so much for helping us on this..

[#10 · Accepted solution]

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.