Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

unable to start name node

avatar
Explorer

Hi Team,

When we initiate the hadoop cluster services using Ambari after enabling kerborose getting authentication failed, logs have attached.

Please help us on this.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.

View solution in original post

9 REPLIES 9

avatar
Master Mentor

@hadoop hdfs

Did you install JCE to unlimited extension?

Have you tried restarting the cluster?

If it still fails can you run the below command

$ su - hdfs 
$ hadoop namenode -recover 

Please revert

avatar
Explorer

Thanks for the reply Geoffrey Shelton,

Did you install JCE to unlimited extension?

--> We have installed jce_policy-8.zip

#unzip -o -j -q <</fullpath>>/jce_policy-8.zip -d /usr/java/jdk1.8.0/jre/lib/security/

Have you tried restarting the cluster?

--Yes


Our hadoop cluster services were running fine without any issues, after enabling kerborose, we are not able to start the services.

When we install kerborose at the last step it will try to restart all the cluster services and some of the services will not come up and operation will fail.

avatar
Explorer

Hi Geoffrey,

We have issued namenode recover command, now all the services are up and running fine ( with kerborose enabled).

We have 3 data nodes and all the data nodes are up, now we are getting below alert on datanode1.

Connection failed to http://rjio-prodcluster-datanode01:1022 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -c /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -w '%{http_code}' http://rjio-prodcluster-datanode01:1022 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpcLoFod 2>/tmp/tmp7T5zRl' returned 7.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to rjio-prodcluster-datanode01:1022; Connection refused

avatar
Explorer

Latest log from datanode1:

Connection failed: [Errno 111] Connection refused to rjio-prodcluster-datanode01:1019

avatar
Master Mentor

@hadoop hdfs

Any updates?

avatar
Explorer

error-log1.pngerror-log2.png@Geoffrey,

We have started the all the services successfully. datanode1 services are started but in ambari UI and hdfs admin report its not showing (showing it as dead node).

We have Two Namenodes & Three data nodes.

Error log and report log have been attached.

report-3.txt

avatar
Master Mentor

@hadoop hdfs
Can you run the below command and then the report to see if the internal view was refreshed. Note dfsadmin are admin scripts so run with caution

$ hdfs dfsadmin -refreshNodes

Please let me know !

avatar
Explorer

@Geoffrey shelton Okot,

After refreshing the nodes we have kill all the processes and started from Ambari, Now all the services are up & running fine.

Thank you so much for helping us on this..

avatar
Master Mentor

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.