Hello,
I have a strange issue and can't figure out why it's
happening. I upgraded our ambari cluster from 2.1.2 to 2.5.2. It all
went well, but now when I login to ambari UI, I do not see my Metrics
charts, it says "No Data". Ambari server log shows errors like these:
05
Jul 2018 08:08:55,516 ERROR [ambari-client-thread-111684]
MetricsRequestHelper:115 - Error getting timeline metrics : Read timed
out
05 Jul 2018 08:08:55,518 ERROR [ambari-client-thread-111684]
MetricsRequestHelper:122 - Cannot connect to collector:
SocketTimeoutException for hbmaster03.pod04.dal09.tealeaf.ibmcloud.com.
Collector service hasn't moved, ports hasn't changed, it's all the same as it was before the upgrade.
From the same ambari server, I can see the data with curl command, for example region count report:
curl -H "Accept: application/json" "http://hbmaster03.pod04.dal09.tealeaf.ibmcloud.com:6188/ws/v1/timeline/metrics?metricNames=regionserver.Server.regionCount._avg&hostname=%&appId=hbase"
I have tried setting these but didn't help:
server.timeline.metrics.cache.connect.timeout.millis=60000
server.timeline.metrics.cache.disabled=true
server.timeline.metrics.cache.interval.read.timeout.millis=60000
server.timeline.metrics.cache.read.timeout.millis=60000
Any advice what to try next?
Many thanks!
Marjana
