Support Questions
Find answers, ask questions, and share your expertise

Datanode threads blocked on synchornized snapshot function of MetricsRegistry class


Datanode threads blocked on synchornized snapshot function of MetricsRegistry class

New Contributor


I posted this on the hadoop users mailing list but thought I will benefit by cross posting to see if users here have come across similar behavior.
I am trying to figure out the reason why our datanode performance degrades after few days and why all the issues fix itself on restart.
We have a lot of remote reads for our HDFS blocks and am noticing a lot of threads are BLOCKED (see stack trace below).
I am wondering why are so many threads trying to build a new snapshot ?
I am not that familiar with this code path and what's the significance of it. Maybe we can turn this off via some configuration and see if that makes the problem go away and then create a JIRA to solve this issue ?
"RMI TCP Connection(584)-" #12963867 daemon prio=5 os_prio=0 tid=0x00007f2b9a58a800 nid=0x1fd3f waiting for monitor entry [0x00007f1ec57d4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(
        - waiting to lock <0x00007f20fc199768> (a org.apache.hadoop.metrics2.lib.MetricsRegistry)
        at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(
        at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getAttribute(
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(
        at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
        at java.lang.reflect.Method.invoke(
        at sun.rmi.server.UnicastServerRef.dispatch(
        at sun.rmi.transport.Transport$
        at sun.rmi.transport.Transport$
        at Method)
        at sun.rmi.transport.Transport.serviceCall(
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$13/ Source)
        at Method)
        at sun.rmi.transport.tcp.TCPTransport$
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
        at java.util.concurrent.ThreadPoolExecutor$
Don't have an account?