Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

Contributor

I am encountering an issue with hive query, where the query takes long time to execute. Usually it takes max 10 mins but sometimes it takes more than 5 hours of time.

Here is log I have captured from the nodemanager where it got stuck:

34,800 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=user5IP=172.20.178.106OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1491671269638_43231CONTAINERID=container_1491671269638_43231_01_000005 2017-04-23 01:07:34,800 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_1491671269638_43231_01_000005 transitioned from LOCALIZING to KILLING 2017-04-23 01:07:34,801 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11_tmp 2017-04-23 01:07:34,803 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(492)) - delete returned false for path: [/hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11_tmp] 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk22/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/10 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk22/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/10_tmp 2017-04-23 01:07:34,803 WARN hdfs.BlockReaderFactory (BlockReaderFactory.java:getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader. java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3497) at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:775) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:692)

Do anyone have any experience with this kinds of issue.

Thanks

Khireswar

1 REPLY 1

Re: Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

@khireswar Kalita

There are high chances that your connection was not configured properly or it might be getting reset.

You might have to hit on the connection configuration.