Support Questions
Find answers, ask questions, and share your expertise

Re: ambari + metrics collector fails

after one hour that metrics collector was up now it down and we see that logs:


at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException: this stream does not support unbuffering.
at org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:233)
... 29 more
2018-06-04 16:19:26,938 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.
Michael-Bronson

Re: ambari + metrics collector fails

Expert Contributor

Michael Bronson

This is a known issue in the HBase version used by AMS in Ambari 2.6.1. Please downgrade AMS version to 2.6.0 using the following steps.

  • Update ambari.repo file on Metrics collector host to point to 2.6.0.0 release
  • yum clean all
  • Stop AMS.
  • yum remove ambari-metrics-collector
  • yum install ambari-metrics-collector
  • Verify version of AMS jar - /usr/lib/ambari-metrics-collector/ambari-metrics-*.jar
  • Start AMS.
  • Update repo file back to 2.6.1 version so that we don't disturb Ambari's setup.

There were minimal changes in AMS from 2.6.0 to 2.6.1. You can also bring back the 2.6.1 versions of ambari-metrics-* jars in /usr/lib/ambari-metrics-collector after the yum downgrade. Meaning, using newest version of AMS jars + older version of HBase.

Re: ambari + metrics collector fails

@Aravindan Vijayan

do you mean this is known issue about the warning that I get - "WARN  [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=48232] io.FSDataInputStreamWrapper: Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So there may be a TCP socket connection left open in CLOSE_WAIT state."  ?
Michael-Bronson