Support Questions
Find answers, ask questions, and share your expertise

Namenode is in Hung state due to deadlock error

Hi,

Today our active NN went down due to thread deadlock's issue. This is the sample from the thread dump. All the IPC server handlers for both 8020 and 8040 were hung due to the below issue. Could you please explain about the lock issue.

org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) Thread 372 (IPC Server handler 83 on 8040): State: WAITING Blocked count: 20620 Waited count: 2798944 Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4391b320 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

1 REPLY 1

Re: Namenode is in Hung state due to deadlock error

Hi,

Could anyone please help me on this asap. We are facing frequent fail overs with this issue.