ambari server not running

[#1]

I am trying to start the Ambari server, the server status is running but not able to view it's UI in the 8080 port. No error logged in the file /var/log/ambari-server/ambari-server.log but there are errors logged in the file /var/log/ambari-agent/ambari-agent.log.

Below are the two errors which are logged in /var/log/ambari-agent/ambari-agent.log,

ERROR 2017-05-08 14:27:20,201 script_alert.py:112 - [Alert][ams_metrics_monitor_process] Failed with result CRITICAL: ['Ambari Monitor is NOT running on http://xxx.YYY.com']

ERROR 2017-05-08 14:27:20,184 script_alert.py:112 - [Alert][yarn_nodemanager_health] Failed with result CRITICAL: ['Connection failed to http://xxx.YYY.com:8042/ws/v1/node/info (Traceback (most recent call last):\n File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py", line 165, in execute\n url_response = urllib2.urlopen(query, timeout=connection_timeout)\n File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen\n return _opener.open(url, data, timeout)\n File "/usr/lib64/python2.6/urllib2.py", line 391, in open\n response = self._open(req, data)\n File "/usr/lib64/python2.6/urllib2.py", line 409, in _open\n \'_open\', req)\n File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain\n result = func(*args)\n File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open\n return self.do_open(httplib.HTTPConnection, req)\n File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open\n raise URLError(err)\nURLError: <urlopen error [Errno 111] Connection refused>\n)']

Getting the below when I try to reach to the ambari-ui:

...Loading...

I am working on this issue for the past 2 days. I am not able to find any related information / resolution in the internet. It will be great if anyone gives some inputs on this issue.

[#2]

To troubleshoot Ambari, use ambari-server log and not ambari-agent log. Default location is /var/log/ambari-server/ambari-server.log

[#3]

Hi @PardeepAs I mentioned there are no error generated in the ambari-server.log file. There are only WARN and INFO messages, please let me know if you need one.

[#4]

above errors related ambari metrics, those are not related your Ambari url loading issue.

Please provide below question :

-running on VMs/AWS/Sandbox/any other environment ? if its HDP sandbox, please reload on VMs.

-Memory details needed on local server ?

Also you have mentioned, no error on /var/log/ambari-server/ambari-server.log...please try to re-start ambari server services and capture the logs...then will review the logs and move on.

[#5]

Hi @Palanivelrajan Chellakutty: Below are answers for your question,

- Yes it is running on a VM.

- Could you please explain what reloading on VM's. The hadoop cluster is a 2 node cluster with 1 gateway node and 1 worker node where I am trying to create testing environment.

- Memory details,

Below is the output of some CPU commands,

--lscpu

Architecture: x86_64

CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian

CPU(s): 4

On-line CPU(s) list: 0-3

Thread(s) per core: 1

Core(s) per socket: 1

Socket(s): 4

NUMA node(s): 1

Vendor ID: XXX

CPU family: 21

Model: 0

Stepping: 2

CPU MHz: 2400.028

BogoMIPS: 4800.05

Hypervisor vendor: YY

Virtualization type: full

L1d cache: 16K

L1i cache: 64K

L2 cache: 2048K

L3 cache: 12288K

--free -m

total used free shared buffers cached

Mem: 32113 10895 21217 0 379 9043

-/+ buffers/cache: 1472 30640

Swap: 5119 0 5119

--uname -mrs

Linux 2.6.32-431.el6.x86_64 x86_64

Please let me know if you need more info.

[#6]

@ajay vembu

Please attach the "/var/log/ambari-server/ambari-server.log" here if possible because the error that you posted in your previous update does not seem to be related to Ambari Server Not starting issue.

[#7]

Hi @Jay SenSharmaAs I mentioned there are no error generated in the ambari-server.log file. There are only WARN and INFO messages, please let me know if you need one.

[#8]

@ajay vembu

Please share the output of the following commands to verify if the ambari server is actually running and listening on port 8080 or not?

# ps -ef | grep `cat /var/run/ambari-server/ambari-server.pid` 
# netstat -tnlpa | grep `cat /var/run/ambari-server/ambari-server.pid`

.

Also as you mentioned that there are only WARN and INFO messages in ambari-server.log so please check and share the "ambari-server.out" file to see if there are any strange issues,

[#9]

Hi @Jay SenSharma:

Below are the details of executing the commands,

ps -ef | grep `cat /var/run/ambari-server/ambari-server.pid`

root 22996 1 0 May08 ? 00:07:03 /usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m -Djava.security.auth.login.config=/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -cp /etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar org.apache.ambari.server.controller.AmbariServer

root 25774 21782 0 08:02 pts/0 00:00:00 grep 22996

netstat -tnlpa | grep `cat /var/run/ambari-server/ambari-server.pid`

tcp 0 0 :::8080 :::* LISTEN 22996/java

tcp 0 0 :::8440 :::* LISTEN 22996/java

tcp 0 0 :::8441 :::* LISTEN 22996/java

tcp 0 0 ::ffff:xx.zz.aaa.yyy:8441 ::ffff:xx.zz.aaa.yyy:53556 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44955 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44926 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44923 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44963 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44964 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44921 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

tcp 0 0 ::ffff:127.0.0.1:44962 ::ffff:127.0.0.1:5432 ESTABLISHED 22996/java

Attaching the output of ambari-server.out.

[#10]

@ajay vembu

Good to see that the following:

1. ambari server is listening properly on port 8080

tcp 0 0 :::8080 :::* LISTEN 22996/java 

2. Ambari's Registration and Heartbeat Port (8441) for Ambari Agents seems to be establish .

tcp 0 0 ::ffff:xx.zz.aaa.yyy:8441 ::ffff:xx.zz.aaa.yyy:53556 ESTABLISHED 22996/java 

3. Handshake Port (8440) for Ambari Agents to Ambari Server is also opened (although no communication seems to be happening)

.

So the problem Does Not seems to be from Ambari Side. So may be we should check the following:

Check-1). From the Client machine (where we are trying to open the ambari UI) are we able to access ambari hostname and connect to the port 8080 (this is to isolate any firewall / Network issue)

# telnet  $AMBARI_HOSTNAME  $AMBARI_PORT

Check-2). Check if the client machine is able to recognize the ambari hostname?

# ping  $AMBARI_HOSTNAME

Check-3). On ambari server host please check if the iptables/firewall is stopped.

# service iptables status
# service iptables stop

.