AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

[#1]

Metrics Collector Process

it shows nothing when i type 'netstat -a|grep 6188'

restart ambari-server or reboot dosen't work either

ambari-server log:

MetricsRequestHelper:114 - Error getting timeline metrics : Connection refused

by the way, although my OS time is right but my log time is wrong

[#2]

@white wartih

When you are trying to start the AMS collector from ambari UI, Do you see any error on the AMS logs?

If the output of the following command is not returning any value, means the AMS collector is not running, So we need to first check the AMS log for any error / exception. Can you pelase share the error/exceptions observed in the ambari-metrics-collector.log ?

netstat -tanlp |grep 6188

.

[#3]

@Jay SenSharma

@Jay SenSharma

ambari-metrics-collector.log:

Exception in thread Thread-947: Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 1082, in run self.function(*self.args, **self.kwargs)

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 45, in process_event self.process_host_collection_event(event)

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 79, in process_host_collection_event metrics.update(self.host_info.get_disk_io_counters())

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 265, in get_disk_io_counters io_counters = psutil.disk_io_counters()

File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk")

RuntimeError: couldn't find any physical disk

thank you very much

[#4]

@white wartih

AMS scripts uses standard Python "psutils" module to find the disk_io_counters. As mentioned in :

https://github.com/apache/ambari/blob/release-2.5.0/ambari-metrics/ambari-metrics-host-monitoring/sr...

File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk") 

The Error which python is detecting is "couldn't find any physical disk"

So can you please check if there are any disk issues? Do you see any issues with the following commands?

# df -h
# du

.

[#5]

@Jay SenSharma

sorry i send you the ambari-metrics-collector.out before

ambari-metrics-collector.txt

# df -h

Filesystem Size Used Avail Use% Mounted on

tank/containers/xdata-0 1.6T 12G 1.6T 1% /

none 492K 4.0K 488K 1% /dev

udev 126G 0 126G 0% /dev/tty

/dev/md0 92G 22G 66G 25% /dev/lxd

none 4.0K 0 4.0K 0% /sys/fs/cgroup none 26G 1.1M 26G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 126G 0 126G 0% /run/shm

none 100M 0 100M 0% /run/user

#du

3750 .

by the way, my another cluster shows that message too,but it works well = =

[#6]

@white wartih

Can you check whether your Ambari-Metrics collector process is running or not. On the same Host where you have installed Metrics collector, Execute the following command,

# netstat -tulpn | grep 6188

If the above command executed with null value, please check the ambari-metrics collector server logs in /var/log/ambari-metrics-collector/ directory. Also try to restart the Ambari Metrics collector and try again.

[#7]

@nshelke

thank a lot!

please check the message above

[#8]

@white wartih

Can you attach your Metrics-Collector logs

[#9]

@nshelke

ambari-metrics-collector.txt

[#10 · Accepted solution]

@white wartih

I see some errors in ambari-metrics-collector logs like,

Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG

Can you check the below link for these errors.

https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html