question Ambari Services Error in Cluster in Support Questions

Ambari Services Error in Cluster

david_yee — Tue, 26 Jan 2016 12:32:59 GMT

Hi All,

When I go to the Cluster's services, most of the services failed. When I tried to restart each services, it took a long time to start it. Help would be appreciated.

services-error-cluster.png

Best Regards

David Yee

Re: Ambari Services Error in Cluster

aervits — Tue, 26 Jan 2016 12:41:32 GMT

@David Yee HDFS is on safe mode and then it will start the rest. This is by design, please post any failed service by showing the logs.

Re: Ambari Services Error in Cluster

david_yee — Tue, 26 Jan 2016 13:38:55 GMT

Hi Artem,

Can you elaborate more on this?

Best Regards

David

Re: Ambari Services Error in Cluster

david_yee — Tue, 26 Jan 2016 13:42:27 GMT

Hi Artem,

When I tried to start all the components. It fails on the "NameNode Start" step and the following error occurs.

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-ip-172-30-1-137.ap-southeast-1.compute.internal.out

Best Regards

David

Re: Ambari Services Error in Cluster

aervits — Tue, 26 Jan 2016 13:43:40 GMT

@David Yee

please post output of /var/log/hadoop/hdfs/hdfs-namenode log

Re: Ambari Services Error in Cluster

david_yee — Tue, 26 Jan 2016 14:00:39 GMT

Hi Artem,

I hope I am sending you the right file.

hadoop-hdfs-namenode-ip-172-30-1-137ap-southeast-1.zip

Best Regards

David Yee

Re: Ambari Services Error in Cluster

nsabharwal — Tue, 26 Jan 2016 17:07:59 GMT

@David Yee

then

1) Bring up the core i,e HDFS, YARN and MapReduce

2) Bring up other services like Hive and so on

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 11:12:50 GMT

[root@ip-172-30-1-137 sbin]# hdfs dfsadmin -safemode leave safemode: Call From ip-172-30-1-137.ap-southeast-1.compute.internal/52.77.231.10 to ip-172-30-1-137.ap-southeast-1.compute.internal:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

I am getting the following error message

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 11:14:21 GMT

[root@ip-172-30-1-137 sbin]# hadoop dfsadmin -safemode leave DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.

safemode: Call From ip-172-30-1-137.ap-southeast-1.compute.internal/52.77.231.10 to ip-172-30-1-137.ap-southeast-1.compute.internal:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

your command return this error that's the reason I am using hdfs instead.

Re: Ambari Services Error in Cluster

nsabharwal — Wed, 27 Jan 2016 11:15:36 GMT

@David Yee Start the namenode process and see what happens.

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 11:21:49 GMT

hadoop-hdfs-namenode-ip-172-30-1-137ap-southeast-1.zip

It is still the same. This is the logfile.

Re: Ambari Services Error in Cluster

nsabharwal — Wed, 27 Jan 2016 11:48:46 GMT

@David Yee

2016-01-26 22:17:29,083 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: ip-172-30-1-137.ap-southeast-1.compute.internal:50075

netstat -anp | grep 50075

it wil give you the process id..kill that process and try again

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 14:35:29 GMT

Hi Neeraj,

I have tried but could not still start it.

Best Regards

David

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 14:42:09 GMT

Hi Neeraj,

Can you help me to check if my hosts file is configured correctly? Just to let you know that I am using AWS EC2 as the server and everytime I restart my server it will give me a new IP address.

Best Regards

David

Re: Ambari Services Error in Cluster

sagarshimpi — Mon, 19 Aug 2019 11:18:27 GMT

Hi @David Yee,

Below are two solution i can suggest to resolve your problem -

1. You can use "Private IP address" instead of public ip address [ie. 52.77.231.10] in /etc/hosts file. No more changes will be required [assuming you have use hostname while registering hosts with ambari]. Changing the hosts file will take an effect to get service started, if not make sure you restart the cluster for changes to take effect. PFA screenshot.

2. You can use Elastic IP for your ec2 instances, so that the ip will never change even if your instance gets restarted. You can see the below link for assigning elastic ip to ec2-instances -

http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-assign-eip.html

[Note: Elastic ip are not free.Make sure you check AWS costing model for the same.]

Re: Ambari Services Error in Cluster

david_yee — Wed, 27 Jan 2016 16:12:52 GMT

Sorry guys managed to finally start the namenode by changing the value to 0.0.0.0. It works perfectly.

Re: Ambari Services Error in Cluster

aervits — Thu, 04 Feb 2016 10:18:04 GMT

@David Yee has this been resolved? Can you provide a solution or accept the best answer?