question Re: How to solve "Connection refused " errors in HDP in Archives of Support Questions (Read Only)

How to solve "Connection refused " errors in HDP

kib — Fri, 16 Sep 2022 10:01:21 GMT

I have been working on Hortonworks Sandbox. The Sandbox is deployed on CentOS 6.7 server. I was, at some point in time, able to access the Web UIs on my browser from remote PC. But most of the services are not accessible except the welcome page on port 8888. When I try to start the services with command line in PuTTY, no success! That is, the services are not running. When I run this command, netstat -nltu , I can't see the service ports in the list of listening ports. How can I make the ports listen for TCP connection. Any walk through on configuration of iptables and port forwarding would also be highly appreciated!

Re: How to solve "Connection refused " errors in HDP

nsabharwal — Mon, 01 Feb 2016 22:17:04 GMT

@Kibrom Gebrehiwot

Start ambari server

Use ambari to start services

Port 8080

Re: How to solve "Connection refused " errors in HDP

aervits — Mon, 01 Feb 2016 22:17:59 GMT

@Kibrom Gebrehiwot you probably shut down the machine abruptly. SSH to the machine, issue shutdown now -r and hope everything comes back. If it doesn't, SSH again, issue "ambari-server start", login to web UI by going to 127.0.0.1:8080 user:admin pass:admin, start services manually. If still doesn't work, import a new sandbox and start from scratch.

Re: How to solve "Connection refused " errors in HDP

nsabharwal — Mon, 01 Feb 2016 22:18:19 GMT

Command

ambari-server start

Re: How to solve "Connection refused " errors in HDP

Shelton — Tue, 02 Feb 2016 05:37:34 GMT

@Kibrom Gebrehiwot

Grab Artem Ervits answer

Re: How to solve "Connection refused " errors in HDP

kib — Tue, 02 Feb 2016 15:16:31 GMT

Thanks @Artem Ervits for your prompt response! I am able to start the ambari server by issuing the command 'ambari-server start' now. But I can't still open Ambari on the browser from my Windows PC. http://<server-name>:8080 shows error: " ERR_CONNECTION_REFUSED". How can I add the HDP service ports to the list of listening ports? I think the issue could be related to port forwarding. How can I forward HDP service ports so that I can access the services from remote machine?

Re: How to solve "Connection refused " errors in HDP

kib — Tue, 02 Feb 2016 15:44:31 GMT

Note: I am working on a network protected by firewall. My Local PC and the remote Sandbox Host are on the same LAN. I am remotely accessing the Sandbox Host (deployed onCentOS ) from my Window PC using PuTTY.

Re: How to solve "Connection refused " errors in HDP

amcbarnett — Tue, 02 Feb 2016 15:46:38 GMT

@Kibrom GebreHiwot

Perhaps your /etc/hosts is wrong...

What's in your /etc/hosts? Is 10.0.225.60 there?
What is the eth0 ip address when you do a ifconfig?

Perhaps your ambari-server really did not start.

What is your ambari-server status after executing

> ambari-server status

If it is not running do

> ambari-server stop
> rm -f /var/run/ambari-server/ambari-server.pid 
> ambari-server start

Do you see an ambari-server process after executing ps -ef | grep ambari-server?
If not what is in the log file /var/log/ambari-server/ambari-server.log?

If you don't see any errors follow the steps here:

https://community.hortonworks.com/articles/6227/sandbox-1270018080-not-accessible.html

If NAT does not work try Bridged and get the ipaddress using ifconfig, restart Ambari-server and use that ip address when you do a http://:8080>:8080

Finally check the firewalls on your host.

Re: How to solve "Connection refused " errors in HDP

kib — Mon, 19 Aug 2019 10:35:20 GMT

Ancil McBarnett Much respect and thanks for your dedication!

I tried all the options, but no luck!

/etc/hosts file looks like this:

eth0 IP address is 10.0.225.60
Ambari Server failed to start. Look at the screen shot
- "Connection refused"
I can see ambari-server process though
running netstat -nltu | grep 8080 returns nothing.

Any thing about port forwarding?

netstat -nltu

8080 is not in the list of listening ports.

Re: How to solve "Connection refused " errors in HDP

Shelton — Tue, 02 Feb 2016 19:29:35 GMT

Seems you have a DB connectivity problem ,can you copy and paste a complete output of the Ambari failed to start log ?

The connection was closed when Java tried to read from it. This can be caused by:

The PostgreSQL server being restarted
The PostgreSQL backend you were connected to being terminated
The PostgreSQL backend you were connected to crashing
Dodgy network connection
Badly behaved stateful firewalls
Idle connections timing out of the NAT connection tables of NAT firewall/routers

... and probably more. Check the PostgreSQL server logs to see if there's anything informative there; also consider doing some network tracing with a tool like Wireshark.

Re: How to solve "Connection refused " errors in HDP

kib — Mon, 19 Aug 2019 10:35:12 GMT

@Geoffrey Shelton Okot

Thanks for your prompt response!

I think the cause for the failure is that the PostgreSQL failed to connect.

service postgresql start

results in : Starting postgresql service: [FAILED]

you can look at the below screen shot too.

vi /var/log/ambari-server/ambari-server.out

Any further fix?

Re: How to solve "Connection refused " errors in HDP

aervits — Tue, 02 Feb 2016 20:01:25 GMT

@Kibrom Gebrehiwot port forwarding is available in network properties of your hypervisor.

Re: How to solve "Connection refused " errors in HDP

kib — Tue, 02 Feb 2016 20:14:02 GMT

@Artem Ervits The Sandbox is deployed on CentOS server (base OS) not on virtual machine. I have no GUI access to the server. I can access it only with PuTTY (remotely).

Re: How to solve "Connection refused " errors in HDP

nsabharwal — Tue, 02 Feb 2016 20:20:13 GMT

@Kibrom Gebrehiwot

I suggest to remove the vm and reimport the image as there are multiple issues. Postgres down and configuration all incorrect.

Re: How to solve "Connection refused " errors in HDP

aervits — Tue, 02 Feb 2016 20:20:24 GMT

@Kibrom Gebrehiwot every hypervisor has CLI take a look at docs how to port forward using command line

Re: How to solve "Connection refused " errors in HDP

Shelton — Tue, 02 Feb 2016 21:18:08 GMT

@Kibrom Gebrehiwot

These are some of the steps you should try before reinstalling your server!

Disable firewall / iptables on the host

Run all the below command like root Execute the below command on the server to disable the firewall, type the following command as the root user to disable firewall for IPv6:

[root@hiroshima01 ~]# service ip6tables stop

[root@hiroshima01 ~]# chkconfig ip6tables off

[root@hiroshima01 ~]# service iptables status IPv4

[root@hiroshima01 etc] # service iptables save

[root@hiroshima01 etc] # service iptables stop

[root@hiroshima01 etc] # chkconfig iptables off

Disable Transparent Huge Pages (THP)

Disable it on the host otherwise the Ambari install will fail

[root@hiroshima01 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

[root@hiroshima01 ~]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

To disable or make these changes persistent across reboots I add this to the bottom of my vi /etc/rc.local

#disable THP at boot time

if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag;

then echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

To validate THP is disabled, I run the below three commands, or any variant you choose from here .

[root@hiroshima01 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag #output should be always madvise [never] [root@hiroshima01 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled # output should be always madvise [never]

Disable SELinux

SELinux must be disabled for Ambari to function. To temporarily disable SELinux, run the following command on each host in your cluster:

[root@hiroshima01 ~]# setenforce 0

Permanently disabling SELinux so that on system reboot it does not restart edit the SELinux config and set SELINUX to disabled. on each host:

root@hiroshima01 ]# vi /etc/selinux/config

Add the below lines

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - SELinux is fully disabled.

SELINUX=disabled

# SELINUXTYPE= type of policy in use. Possible values are:

# targeted - Only targeted network daemons are protected.

# strict - Full SELinux protection.

SELINUXTYPE=targeted

Configure the NTPD services

You must setup the NTPD server on CentOS to successfully implement Ambari follow the below steps to install and start the NTPD server. As the root user.

To install ntpdate

[root@hiroshima01 ~]# yum install ntp ntpdate

Turn on the service:

[root@hiroshima01 ~]# chkconfig ntpd on

Synchronize the system clock with 0.pool.ntp.org server

[root@hiroshima01 bin]# ntpdate pool.ntp.org

2 Jan 21:42:49 ntpdate[5101]: 31.3.135.236 rate limit response from server.

2 Jan 21:43:56 ntpdate[5101]: step time server 212.51.144.44 offset 67.520940 sec

Check the NTPD services configure it start at system boot

[root@hiroshima01 bin]# chkconfig --list ntpd ntpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Start the NTP server.

The following will continuously adjust system time from upstream NTP server. No need to run ntpdate:

[root@hiroshima01 ~]# /etc/init.d/ntpd start or

[root@hiroshima01 ~]# service ntpd start

Re: How to solve "Connection refused " errors in HDP

Shelton — Tue, 02 Feb 2016 21:22:37 GMT

Did you follow these steps during your setup

Postgres for Ambari steps

Re: How to solve "Connection refused " errors in HDP

kib — Tue, 02 Feb 2016 21:54:30 GMT

@Geoffrey Shelton Okot

I followed all the above steps but Ambari server and PostgreSQL can not start. Hue also failed to start.

> /etc/init.d/hue start # failed

The PostreSQL is embedded by default with HDP Sandbox 2.3. So, I did not do any configuration and it was working fine before some time.

Do I have to go for reinstalling the Sandbox?

Re: How to solve "Connection refused " errors in HDP

kib — Tue, 02 Feb 2016 22:01:34 GMT

@Neeraj Sabharwal

Ambari server can't start at all!

Re: How to solve "Connection refused " errors in HDP

Shelton — Tue, 02 Feb 2016 22:14:02 GMT

@Kibrom Gebrehiwot Re-install should be the last resort because these are real conditions you will meet so start to diagnose and get the sandbox up and running would be a great feat !

How is you network configuration for the Sandbox?

Do backup your /etc/hosts and create a new hosts with the below entry

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.225.60 Sandbox.hortonworks.com Sandbox

Set your network to DHCP and restart the sandbox