Support Questions
Find answers, ask questions, and share your expertise

Datanode Breaching Xceiver count

Datanode Breaching Xceiver count

New Contributor

In a 900 Nodes cluster, we observed the breach of xceiver counts on a particular set of datanodes and all those were repeated.
Xceiver count may increase based on the demand for data by the HDFS Client. But why on a particular set of nodes is not described.

And here is stack trace of the datanode

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff; min-height: 15.0px} span.s1 {font-variant-ligatures: no-common-ligatures}

"DataXceiver for client DFSClient_attempt_1526704594842_1959147_m_005330_2_111824500_1 at /x.x.x.x:38597 [Waiting for operation #2]" #103160733 daemon prio=5 os_prio=0 tid=0x00007f2db8f5a800 nid=0x515f runnable [0x00007f2d8db18000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

- locked <0x0000000718125d78> (a sun.nio.ch.Util$2)

- locked <0x0000000718125ce8> (a java.util.Collections$UnmodifiableSet)

- locked <0x000000078ff1e468> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)

at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

- locked <0x00000005c3551948> (a java.io.BufferedInputStream)

at java.io.DataInputStream.readShort(DataInputStream.java:312)

at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

- None

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff; min-height: 15.0px} span.s1 {font-variant-ligatures: no-common-ligatures}

"DataXceiver for client DFSClient_attempt_1526704594842_1959147_m_005150_2_-764016440_1 at /x.x.x.x:47536 [Waiting for operation #2]" #103160732 daemon prio=5 os_prio=0 tid=0x00007f2db9ee5000 nid=0x515e runnable [0x00007f2cc5493000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

- locked <0x0000000718c01780> (a sun.nio.ch.Util$2)

- locked <0x0000000718c01768> (a java.util.Collections$UnmodifiableSet)

- locked <0x000000078ff6b9a8> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)

at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

- locked <0x00000005c35218f8> (a java.io.BufferedInputStream)

at java.io.DataInputStream.readShort(DataInputStream.java:312)

at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

- None

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo; color: #000000; background-color: #ffffff; min-height: 15.0px} span.s1 {font-variant-ligatures: no-common-ligatures}

"DataXceiver for client DFSClient_attempt_1526704594842_1959158_m_010809_1_-1736176183_1 at /x.x.x.x:17139 [Waiting for operation #2]" #103160730 daemon prio=5 os_prio=0 tid=0x00007f2db8eec000 nid=0x515c runnable [0x00007f2d55add000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

- locked <0x00000007194967f8> (a sun.nio.ch.Util$2)

- locked <0x0000000719496768> (a java.util.Collections$UnmodifiableSet)

- locked <0x000000078ff74d50> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)

at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

- locked <0x00000005c34a4d00> (a java.io.BufferedInputStream)

at java.io.DataInputStream.readShort(DataInputStream.java:312)

at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)

at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)

at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:

- None