Support Questions

Find answers, ask questions, and share your expertise

Ambari Services Error in Cluster

Explorer

Hi All,

When I go to the Cluster's services, most of the services failed. When I tried to restart each services, it took a long time to start it. Help would be appreciated.

services-error-cluster.png

Best Regards

David Yee

1 ACCEPTED SOLUTION

Expert Contributor

Hi @David Yee,

Below are two solution i can suggest to resolve your problem -

1. You can use "Private IP address" instead of public ip address [ie. 52.77.231.10] in /etc/hosts file. No more changes will be required [assuming you have use hostname while registering hosts with ambari]. Changing the hosts file will take an effect to get service started, if not make sure you restart the cluster for changes to take effect. PFA screenshot.

1579-ec2-private-ip-address.png

2. You can use Elastic IP for your ec2 instances, so that the ip will never change even if your instance gets restarted. You can see the below link for assigning elastic ip to ec2-instances -

http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-assign-eip.html

[Note: Elastic ip are not free.Make sure you check AWS costing model for the same.]

View solution in original post

16 REPLIES 16

Mentor

@David Yee HDFS is on safe mode and then it will start the rest. This is by design, please post any failed service by showing the logs.

Explorer

Hi Artem,

When I tried to start all the components. It fails on the "NameNode Start" step and the following error occurs.

resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start namenode'' returned 1. starting namenode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-namenode-ip-172-30-1-137.ap-southeast-1.compute.internal.out

Best Regards

David

Mentor
@David Yee

please post output of /var/log/hadoop/hdfs/hdfs-namenode log

Explorer

Hi Artem,

I hope I am sending you the right file.

hadoop-hdfs-namenode-ip-172-30-1-137ap-southeast-1.zip

Best Regards

David Yee

Explorer

Hi Artem,

Can you elaborate more on this?

Best Regards

David

@David Yee

Login to namenode host and run this hadoop dfsadmin -safemode leave

then

1) Bring up the core i,e HDFS, YARN and MapReduce

2) Bring up other services like Hive and so on

Explorer

[root@ip-172-30-1-137 sbin]# hdfs dfsadmin -safemode leave safemode: Call From ip-172-30-1-137.ap-southeast-1.compute.internal/52.77.231.10 to ip-172-30-1-137.ap-southeast-1.compute.internal:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

I am getting the following error message

@David Yee Start the namenode process and see what happens.

Explorer

[root@ip-172-30-1-137 sbin]# hadoop dfsadmin -safemode leave DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.

safemode: Call From ip-172-30-1-137.ap-southeast-1.compute.internal/52.77.231.10 to ip-172-30-1-137.ap-southeast-1.compute.internal:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

your command return this error that's the reason I am using hdfs instead.

Explorer

hadoop-hdfs-namenode-ip-172-30-1-137ap-southeast-1.zip

It is still the same. This is the logfile.

@David Yee

2016-01-26 22:17:29,083 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode. java.net.BindException: Port in use: ip-172-30-1-137.ap-southeast-1.compute.internal:50075

netstat -anp | grep 50075

it wil give you the process id..kill that process and try again

Explorer

Hi Neeraj,

I have tried but could not still start it.

Best Regards

David

Explorer

Hi Neeraj,

Can you help me to check if my hosts file is configured correctly? Just to let you know that I am using AWS EC2 as the server and everytime I restart my server it will give me a new IP address.

Best Regards

David

Expert Contributor

Hi @David Yee,

Below are two solution i can suggest to resolve your problem -

1. You can use "Private IP address" instead of public ip address [ie. 52.77.231.10] in /etc/hosts file. No more changes will be required [assuming you have use hostname while registering hosts with ambari]. Changing the hosts file will take an effect to get service started, if not make sure you restart the cluster for changes to take effect. PFA screenshot.

1579-ec2-private-ip-address.png

2. You can use Elastic IP for your ec2 instances, so that the ip will never change even if your instance gets restarted. You can see the below link for assigning elastic ip to ec2-instances -

http://docs.aws.amazon.com/AmazonVPC/latest/GettingStartedGuide/getting-started-assign-eip.html

[Note: Elastic ip are not free.Make sure you check AWS costing model for the same.]

Explorer

Sorry guys managed to finally start the namenode by changing the value to 0.0.0.0. It works perfectly.

Mentor

@David Yee has this been resolved? Can you provide a solution or accept the best answer?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.