Not able to start service

[#1]

ambari.pngambari-1.pngWe are facing this problem since last week and aren't able to start services. We are trying to install HDP(2.4) on a 3 node cluster. We were able to download and install all the packages via ambari GUI but it failed to start the services in the last step of installation. Then I tried to start service manually. Some of the services are started. While checking we found below error. Can any one tell me the issue?

Errno 111] Connection refused to XXX.yyy.com:21000 Connection failed to http://XXX.yyy.com:21000 (urlopen error [Errno 111] Connection refused)

Connection failed to http://XXX.yyy.com:50070 (urlopen error [Errno 111]

Connection refused) Connection failed to http://XXX.yyy.com:8088 ( )

All hosts are pingable from each other in the cluster. .There are entry in /etc/host file.

While checking http://<hostname>:50070 , it gives me below error

Connection refused

The server does not have a DNS entry. Perhaps there is a misspelling in the server name, or the server no longer exists. Double-check the name and try again.

[#2]

can you please check the op log of one of the service and see what is the error there ?

[#3]

error.txt

We are getting connection refused error.Please see below error for oozie. Also attaching error log

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://abc00675971901.com:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs' 1>/tmp/tmplxoIGV 2>/tmp/tmp0bDc7c' returned 7. curl: (7) Failed to connect to abc00675971901.com:50070 port 50070: Connection refused

[#4]

@Rajib Mandal

Connection refused usually mean either service is not running or firewall is blocking the port.

If namenode process is running on hdfs namenode host, please verify that you have correct firewall rules to allow required port.

You can try to disable iptables and selinux on the hosts.

#service iptables stop

[#5]

@Rajib Mandal

port 50070 is namenode UI port and 8088 is RM UI which should be accessible if namenode and yarn services are up.

Please check if you can connect to port using telnet from ambari server

#telnet <hostnameInError> 50070

Also on server where namenode is running verify if the port is in listen state, if it is not then namenode service has some problem(you can check the namenode logs to see if the the service exited for any reason).

#netstat -ntalp | grep 50070

[#6]

ntlap.pngI am getting connection refused error while running #telnet <hostnameInError> 50070

Also find attached detail for #netstat -ntalp | grep 50070

[#7]

@Rajib Mandal

The below services are NOT started.

MapReduce2 
Hive 
Hbase 
Oozie 
Falcon 
Storm 
Altlas 
Kafka 

Pinging the hosts isn't no big deal.

Questions

Did you configure the passwordless logon from the Ambari to the other 2 nodes? 
Disable firewall /Iptables or Ip6tables? 
Disable SeLinux.? 
Disables THP ?
 Configure NTPD? 

And lastly did you configure the Database for Hive or Oozie if you ain't using the default derby? 

Can you post here these logs

Ambari Server logs are found at /var/log/ambari-server/ambari-server.log 
Ambari Agent logs are found at /var/log/ambari-agent/ambari-agent.log .

[#8]

Please find my answer

1. Did you configure the passwordless logon from the Ambari to the other 2 nodes?
2. yes
3. Disable firewall /IptablesorIp6tables?
4. yes
5. DisableSeLinux.?
6. yes
7. Disables THP ?
8. yes
9. Configure NTPD?
10. yesambari-agent.pngambari-server.png
11. And lastly did you configure the Database for Hive or Oozie if you ain't using the default derby?
12. we have created separate database for Hive and OoZie

[#9]

@Rajib Mandal

Check the data directories to confirm they are mounted as expected.unmounted data directories on the root partition may indicate that a data directory is writing to the root partition, which is undesirable.

Error Datanode_unmounted_data_dir

Also check permissions.

Can you post the

 
/var/log/ambari-metrics-collector/ambari-metrics-collector.log

[#10]

ambari-metrics-collector.pngMetrics collector log attached. Also /data/hadoop/hdfs/data is not available.For which below error Datanode_unmounted_data_dir occurs