Created 07-16-2018 07:54 AM
My env: HDP 2.6.3, multinode, not kerberized
I have realize in /home/log/ambari-metrics-collector/hbase-root-master-analitik01.log
I see the following warning repeatedly
WARN [pool-40-thread-1] io.FSDataInputStreamWrapper: Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream . So there may be a TCP socket connection left open in CLOSE_WAIT state. java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.unbuffer(FSDataInputStreamWrapper.java:263) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.unbufferStream(HFileBlock.java:1788) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.unbufferStream(HFileReaderV2.java:1403) at org.apache.hadoop.hbase.io.hfile.AbstractHFileReader$Scanner.close(AbstractHFileReader.java:343) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.close(StoreFileScanner.java:252) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:196) at org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:268) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:815) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:792) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:621) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5791) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5778) at org.apache.phoenix.coprocessor.MetaDataRegionObserver$BuildIndexScheduleTask.run(MetaDataRegionObserver.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsupportedOperationException: this stream does not support unbuffering. at org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:233) ... 29 more
How to stop it?
Created 07-16-2018 08:08 AM
The issue which is causing too many CLOSE_WAIT connections in AMS as following is already addressed in Ambari 2.6.2.2 So please try to upgrade your Ambari to 2.6.2.2 (if you are using Ambari 2.6.1)
WARN [pool-40-thread-1] io.FSDataInputStreamWrapper: Failed to invoke 'unbuffer' method in class class org.apache.hadoop.fs.FSDataInputStream .
.
Created 07-16-2018 12:05 PM
Hi @Jay Kumar SenSharma thank you. I use Ambari 2.6.1. Once I upgrade to 2.6.2 I let you know. Thanks again.