question Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host) in Archives of Support Questions (Read Only)

ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

vishnuhr — Fri, 29 Jan 2016 12:08:47 GMT

When i login to the ui, metrics summary for hbase is not shown. for the rest like hdfs/storm the metrics show up.

collector and ambari server are on the same host.

The exception i get in server log is:

--------------------------------

DEBUG [qtp-ambari-client-26 - /api/v1/clusters/cluster_name/services/HBASE/components/HBASE_REGIONSERVER?_=1454039202858] MetricsRequestHelper:116 - Error getting timeline metrics : Read timed out

java.net.SocketTimeoutException: Read timed out

at java.net.SocketInputStream.socketRead0(Native Method)

--------------------------------

any help would be appreciated.

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

nsabharwal — Fri, 29 Jan 2016 12:12:14 GMT

@vishnu rao

Make sure that HBase and RS are up .

Restart the AMS

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

rahulpathak109 — Fri, 29 Jan 2016 13:45:08 GMT

Also check for firewall/iptables issues.

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

vishnuhr — Fri, 29 Jan 2016 14:17:24 GMT

hi Neeraj,

The Hbase ms and rs are up. i am able to see the summary. but not the graphs.

I seemed to have worked around the problem by deleting a widget. (read latency widget) and everything appeared.

even though 1 latency widget is not there.

i added a graph via ambari-grafana for read latency and the metric is now available there

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

vishnuhr — Sat, 30 Jan 2016 08:33:15 GMT

Hi rahul.

The collector and server are on the same host and so is ambari grafana

As I mentioned, removal of a widget made everything else appear. I have added a graph in grafana for the metric widget I removed

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

aervits — Sat, 30 Jan 2016 20:10:42 GMT

@vishnu rao

Firewall can affect monitors sending data to collector so please make sure all servers are allowed

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

nsabharwal — Sat, 30 Jan 2016 21:00:55 GMT

@vishnu rao That sounds like a good work around. I am focusing on this message "java.net.SocketTimeoutException: Read timed out"

Red time out can occur when there is network latency. Nice explanation

For example: Let's say everything is working fine and load in the node is very high , connection is trying to read the data for metric and it cant because there is no more room to do the processing.

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

vishnuhr — Sun, 31 Jan 2016 09:56:47 GMT

aye. agree. seems to be a load issue, but its kinda strange. will debug more.

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

aervits — Sun, 31 Jan 2016 10:11:26 GMT

@vishnu rao

follow the links in this page for tuning, known issues and troubleshooting Link

Re: ambari server unable to connect to collector for hbase metrics - Read SocketTimeoutException (both on same host)

nsabharwal — Sun, 31 Jan 2016 10:53:05 GMT

@vishnu rao Ok. Keep us posted. If it's load issue or network overloaded then I guess we do have a root cause.