Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

Solved Go to solution
Highlighted

AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

New Contributor

Metrics Collector Process

it shows nothing when i type 'netstat -a|grep 6188'

restart ambari-server or reboot dosen't work either

ambari-server log:

MetricsRequestHelper:114 - Error getting timeline metrics : Connection refused

by the way, although my OS time is right but my log time is wrong

1 ACCEPTED SOLUTION

Accepted Solutions

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

@white wartih

I see some errors in ambari-metrics-collector logs like,

Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG

Can you check the below link for these errors.

https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html

11 REPLIES 11

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

Super Mentor

@white wartih

When you are trying to start the AMS collector from ambari UI, Do you see any error on the AMS logs?

If the output of the following command is not returning any value, means the AMS collector is not running, So we need to first check the AMS log for any error / exception. Can you pelase share the error/exceptions observed in the ambari-metrics-collector.log ?

netstat -tanlp |grep 6188

.

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

New Contributor

@Jay SenSharma

@Jay SenSharma

ambari-metrics-collector.log:

Exception in thread Thread-947: Traceback (most recent call last):

File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 1082, in run self.function(*self.args, **self.kwargs)

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 45, in process_event self.process_host_collection_event(event)

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/metric_collector.py", line 79, in process_host_collection_event metrics.update(self.host_info.get_disk_io_counters())

File "/usr/lib/python2.6/site-packages/resource_monitoring/core/host_info.py", line 265, in get_disk_io_counters io_counters = psutil.disk_io_counters()

File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk")

RuntimeError: couldn't find any physical disk

thank you very much

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

Super Mentor

@white wartih

AMS scripts uses standard Python "psutils" module to find the disk_io_counters. As mentioned in :

https://github.com/apache/ambari/blob/release-2.5.0/ambari-metrics/ambari-metrics-host-monitoring/sr...

File "/usr/lib/python2.6/site-packages/resource_monitoring/psutil/build/lib.linux-x86_64-2.7/psutil/__init__.py", line 1726, in disk_io_counters raise RuntimeError("couldn't find any physical disk") 

The Error which python is detecting is "couldn't find any physical disk"

So can you please check if there are any disk issues? Do you see any issues with the following commands?

# df -h
# du

.

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

New Contributor

@Jay SenSharma

sorry i send you the ambari-metrics-collector.out before

ambari-metrics-collector.txt

# df -h

Filesystem Size Used Avail Use% Mounted on

tank/containers/xdata-0 1.6T 12G 1.6T 1% /

none 492K 4.0K 488K 1% /dev

udev 126G 0 126G 0% /dev/tty

/dev/md0 92G 22G 66G 25% /dev/lxd

none 4.0K 0 4.0K 0% /sys/fs/cgroup none 26G 1.1M 26G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 126G 0 126G 0% /run/shm

none 100M 0 100M 0% /run/user

#du

3750 .

by the way, my another cluster shows that message too,but it works well = =

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

@white wartih

Can you check whether your Ambari-Metrics collector process is running or not. On the same Host where you have installed Metrics collector, Execute the following command,

# netstat -tulpn | grep 6188

If the above command executed with null value, please check the ambari-metrics collector server logs in /var/log/ambari-metrics-collector/ directory. Also try to restart the Ambari Metrics collector and try again.

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

New Contributor

@nshelke

thank a lot!

please check the message above

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

@white wartih

Can you attach your Metrics-Collector logs

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

New Contributor

Re: AMS:Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

@white wartih

I see some errors in ambari-metrics-collector logs like,

Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG

Can you check the below link for these errors.

https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html