How to solve "Connection refused " errors in HDP

[#1]

I have been working on Hortonworks Sandbox. The Sandbox is deployed on CentOS 6.7 server. I was, at some point in time, able to access the Web UIs on my browser from remote PC. But most of the services are not accessible except the welcome page on port 8888. When I try to start the services with command line in PuTTY, no success! That is, the services are not running. When I run this command, netstat -nltu , I can't see the service ports in the list of listening ports. How can I make the ports listen for TCP connection. Any walk through on configuration of iptables and port forwarding would also be highly appreciated!

[#2]

@Kibrom Gebrehiwot every hypervisor has CLI take a look at docs how to port forward using command line

[#3]

@Kibrom GebreHiwot

Perhaps your /etc/hosts is wrong...

• What's in your /etc/hosts? Is 10.0.225.60 there?
• What is the eth0 ip address when you do a ifconfig?

Perhaps your ambari-server really did not start.

What is your ambari-server status after executing

> ambari-server status

If it is not running do

> ambari-server stop
> rm -f /var/run/ambari-server/ambari-server.pid 
> ambari-server start

• Do you see an ambari-server process after executing ps -ef | grep ambari-server?
• If not what is in the log file /var/log/ambari-server/ambari-server.log?

If you don't see any errors follow the steps here:

https://community.hortonworks.com/articles/6227/sandbox-1270018080-not-accessible.html

If NAT does not work try Bridged and get the ipaddress using ifconfig, restart Ambari-server and use that ip address when you do a http://:8080>:8080

Finally check the firewalls on your host.

[#4]

Ancil McBarnett Much respect and thanks for your dedication!

I tried all the options, but no luck!

• /etc/hosts file looks like this:

• eth0 IP address is 10.0.225.60
• Ambari Server failed to start. Look at the screen shot
  • "Connection refused"
• I can see ambari-server process though
• running netstat -nltu | grep 8080 returns nothing.

Any thing about port forwarding?

netstat -nltu

8080 is not in the list of listening ports.

[#5]

Seems you have a DB connectivity problem ,can you copy and paste a complete output of the Ambari failed to start log ?

The connection was closed when Java tried to read from it. This can be caused by:

• The PostgreSQL server being restarted
• The PostgreSQL backend you were connected to being terminated
• The PostgreSQL backend you were connected to crashing
• Dodgy network connection
• Badly behaved stateful firewalls
• Idle connections timing out of the NAT connection tables of NAT firewall/routers

... and probably more. Check the PostgreSQL server logs to see if there's anything informative there; also consider doing some network tracing with a tool like Wireshark.

[#6]

@Geoffrey Shelton Okot

Thanks for your prompt response!

I think the cause for the failure is that the PostgreSQL failed to connect.

service postgresql start

results in : Starting postgresql service: [FAILED]

you can look at the below screen shot too.

vi /var/log/ambari-server/ambari-server.out

Any further fix?

[#7]

@Kibrom Gebrehiwot

I suggest to remove the vm and reimport the image as there are multiple issues. Postgres down and configuration all incorrect.

[#8]

@Kibrom Gebrehiwot

These are some of the steps you should try before reinstalling your server!

Disable firewall / iptables on the host

Run all the below command like root Execute the below command on the server to disable the firewall, type the following command as the root user to disable firewall for IPv6:

[root@hiroshima01 ~]# service ip6tables stop

[root@hiroshima01 ~]# chkconfig ip6tables off

[root@hiroshima01 ~]# service iptables status IPv4

[root@hiroshima01 etc] # service iptables save

[root@hiroshima01 etc] # service iptables stop

[root@hiroshima01 etc] # chkconfig iptables off

Disable Transparent Huge Pages (THP)

Disable it on the host otherwise the Ambari install will fail

[root@hiroshima01 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

[root@hiroshima01 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

To disable or make these changes persistent across reboots I add this to the bottom of my vi /etc/rc.local

#disable THP at boot time

if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

fi

if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag;

then echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

fi

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

[root@hiroshima01 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag #output should be always madvise [never] [root@hiroshima01 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled # output should be always madvise [never]

Disable SELinux

SELinux must be disabled for Ambari to function. To temporarily disable SELinux, run the following command on each host in your cluster:

[root@hiroshima01 ~]# setenforce 0

Permanently disabling SELinux so that on system reboot it does not restart edit the SELinux config and set SELINUX to disabled. on each host:

root@hiroshima01 ]# vi /etc/selinux/config

Add the below lines

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - SELinux is fully disabled.

SELINUX=disabled

# SELINUXTYPE= type of policy in use. Possible values are:

# targeted - Only targeted network daemons are protected.

# strict - Full SELinux protection.

SELINUXTYPE=targeted

Configure the NTPD services

You must setup the NTPD server on CentOS to successfully implement Ambari follow the below steps to install and start the NTPD server. As the root user.

To install ntpdate

[root@hiroshima01 ~]# yum install ntp ntpdate

Turn on the service:

[root@hiroshima01 ~]# chkconfig ntpd on

Synchronize the system clock with 0.pool.ntp.org server

[root@hiroshima01 bin]# ntpdate pool.ntp.org

2 Jan 21:42:49 ntpdate[5101]: 31.3.135.236 rate limit response from server.

2 Jan 21:43:56 ntpdate[5101]: step time server 212.51.144.44 offset 67.520940 sec

Check the NTPD services configure it start at system boot

[root@hiroshima01 bin]# chkconfig --list ntpd ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Start the NTP server.

The following will continuously adjust system time from upstream NTP server. No need to run ntpdate:

[root@hiroshima01 ~]# /etc/init.d/ntpd start or

[root@hiroshima01 ~]# service ntpd start

[#9]

Did you follow these steps during your setup

Postgres for Ambari steps

[#10]

@Geoffrey Shelton Okot

I followed all the above steps but Ambari server and PostgreSQL can not start. Hue also failed to start.

> /etc/init.d/hue start # failed

The PostreSQL is embedded by default with HDP Sandbox 2.3. So, I did not do any configuration and it was working fine before some time.

Do I have to go for reinstalling the Sandbox?

[#11]

@Kibrom Gebrehiwot Re-install should be the last resort because these are real conditions you will meet so start to diagnose and get the sandbox up and running would be a great feat !

How is you network configuration for the Sandbox?

Do backup your /etc/hosts and create a new hosts with the below entry

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.225.60 Sandbox.hortonworks.com Sandbox

Set your network to DHCP and restart the sandbox