Reply
Explorer
Posts: 8
Registered: ‎07-18-2016

Hbase regionserver crash due to java.net.SocketTimeoutException:60000 millis timeout

[ Edited ]

Hi,

 

we got java.net.SocketTimeoutException: 60000 millis timeout when we are loading heavy read/write operation in hbase server. It is three node cluster.

Error 1:

java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connect

ed local=/x.x.x.x:50010 remote=/x.x.x.x:32785]

        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)

        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)

        at java.io.DataInputStream.read(DataInputStream.java:149)

        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)

        at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)

       

Error 2:

 

ERROR [regionserver/x.x.x.x:60020] wal.ProtobufLogWriter: Got IOException whle writing trailer
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hbase/oldWALs/Xyz%2C60020%2C1508292160534..meta.1509145944162.meta (inode 15274976): File is not open for writing. [Lease. Holder: DFSClient_NONMAPREDUCE_-926180906_1, pendingcreates: 3]
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3620)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3520)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:702)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAdditionalDatanode(AuthorizationProviderProxyClientProtocol.java:228)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

 

Anyone has experience this? Any suggestion to avoid this type of errors in hbase?

 

Thanks,

Srinivas

 

 

 

Highlighted
Cloudera Employee
Posts: 29
Registered: ‎08-16-2016

Re: Hbase regionserver crash due to java.net.SocketTimeoutException:60000 millis timeout

It looks like a network issue, or DataNodes suffering from long Java GC pauses.

Announcements