question Re: unable to start name node in Archives of Support Questions (Read Only)

unable to start name node

hdfsissue — Tue, 13 Mar 2018 17:05:35 GMT

Hi Team,

When we initiate the hadoop cluster services using Ambari after enabling kerborose getting authentication failed, logs have attached.

Please help us on this.

Re: unable to start name node

Shelton — Tue, 13 Mar 2018 17:56:24 GMT

@hadoop hdfs

Did you install JCE to unlimited extension?

Have you tried restarting the cluster?

If it still fails can you run the below command

$ su - hdfs 
$ hadoop namenode -recover

Please revert

Re: unable to start name node

hdfsissue — Wed, 14 Mar 2018 12:36:50 GMT

Thanks for the reply Geoffrey Shelton,

Did you install JCE to unlimited extension?

--> We have installed jce_policy-8.zip

#unzip -o -j -q <</fullpath>>/jce_policy-8.zip -d /usr/java/jdk1.8.0/jre/lib/security/

Have you tried restarting the cluster?

--Yes

Our hadoop cluster services were running fine without any issues, after enabling kerborose, we are not able to start the services.

When we install kerborose at the last step it will try to restart all the cluster services and some of the services will not come up and operation will fail.

Re: unable to start name node

hdfsissue — Wed, 14 Mar 2018 14:58:57 GMT

Hi Geoffrey,

We have issued namenode recover command, now all the services are up and running fine ( with kerborose enabled).

We have 3 data nodes and all the data nodes are up, now we are getting below alert on datanode1.

Connection failed to http://rjio-prodcluster-datanode01:1022 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -c /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -w '%{http_code}' http://rjio-prodcluster-datanode01:1022 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpcLoFod 2>/tmp/tmp7T5zRl' returned 7.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to rjio-prodcluster-datanode01:1022; Connection refused

Re: unable to start name node

hdfsissue — Wed, 14 Mar 2018 17:18:22 GMT

Latest log from datanode1:

Connection failed: [Errno 111] Connection refused to rjio-prodcluster-datanode01:1019

Re: unable to start name node

Shelton — Thu, 15 Mar 2018 01:43:35 GMT

@hadoop hdfs

Any updates?

Re: unable to start name node

hdfsissue — Thu, 15 Mar 2018 13:07:11 GMT

error-log1.png error-log2.png@Geoffrey,

We have started the all the services successfully. datanode1 services are started but in ambari UI and hdfs admin report its not showing (showing it as dead node).

We have Two Namenodes & Three data nodes.

Error log and report log have been attached.

report-3.txt

Re: unable to start name node

Shelton — Fri, 16 Mar 2018 04:35:00 GMT

@hadoop hdfs
Can you run the below command and then the report to see if the internal view was refreshed. Note dfsadmin are admin scripts so run with caution

$ hdfs dfsadmin -refreshNodes

Please let me know !

Re: unable to start name node

hdfsissue — Mon, 19 Mar 2018 12:27:26 GMT

@Geoffrey shelton Okot,

After refreshing the nodes we have kill all the processes and started from Ambari, Now all the services are up & running fine.

Thank you so much for helping us on this..

Re: unable to start name node

Shelton — Mon, 19 Mar 2018 13:39:12 GMT

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.