Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to start name node

Solved Go to solution
Highlighted

unable to start name node

Explorer

Hi Team,

When we initiate the hadoop cluster services using Ambari after enabling kerborose getting authentication failed, logs have attached.

Please help us on this.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: unable to start name node

Mentor

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.

View solution in original post

9 REPLIES 9
Highlighted

Re: unable to start name node

Mentor

@hadoop hdfs

Did you install JCE to unlimited extension?

Have you tried restarting the cluster?

If it still fails can you run the below command

$ su - hdfs 
$ hadoop namenode -recover 

Please revert

Highlighted

Re: unable to start name node

Explorer

Thanks for the reply Geoffrey Shelton,

Did you install JCE to unlimited extension?

--> We have installed jce_policy-8.zip

#unzip -o -j -q <</fullpath>>/jce_policy-8.zip -d /usr/java/jdk1.8.0/jre/lib/security/

Have you tried restarting the cluster?

--Yes


Our hadoop cluster services were running fine without any issues, after enabling kerborose, we are not able to start the services.

When we install kerborose at the last step it will try to restart all the cluster services and some of the services will not come up and operation will fail.

Highlighted

Re: unable to start name node

Explorer

Hi Geoffrey,

We have issued namenode recover command, now all the services are up and running fine ( with kerborose enabled).

We have 3 data nodes and all the data nodes are up, now we are getting below alert on datanode1.

Connection failed to http://rjio-prodcluster-datanode01:1022 (Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -c /var/lib/ambari-agent/tmp/cookies/63785053-97bc-4585-9177-af9ca9793ea0 -w '%{http_code}' http://rjio-prodcluster-datanode01:1022 --connect-timeout 5 --max-time 7 -o /dev/null 1>/tmp/tmpcLoFod 2>/tmp/tmp7T5zRl' returned 7.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to rjio-prodcluster-datanode01:1022; Connection refused

Re: unable to start name node

Explorer

Latest log from datanode1:

Connection failed: [Errno 111] Connection refused to rjio-prodcluster-datanode01:1019
Highlighted

Re: unable to start name node

Mentor

@hadoop hdfs

Any updates?

Highlighted

Re: unable to start name node

Explorer

error-log1.pngerror-log2.png@Geoffrey,

We have started the all the services successfully. datanode1 services are started but in ambari UI and hdfs admin report its not showing (showing it as dead node).

We have Two Namenodes & Three data nodes.

Error log and report log have been attached.

report-3.txt

Highlighted

Re: unable to start name node

Mentor

@hadoop hdfs
Can you run the below command and then the report to see if the internal view was refreshed. Note dfsadmin are admin scripts so run with caution

$ hdfs dfsadmin -refreshNodes

Please let me know !

Highlighted

Re: unable to start name node

Explorer

@Geoffrey shelton Okot,

After refreshing the nodes we have kill all the processes and started from Ambari, Now all the services are up & running fine.

Thank you so much for helping us on this..

Highlighted

Re: unable to start name node

Mentor

@hadoop hdfs

Fantastic, it's always good to know that a solution provided helped.

Could you Accept the answer by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for this kind of errors.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here