Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

Highlighted

Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

Contributor

I am encountering an issue with hive query, where the query takes long time to execute. Usually it takes max 10 mins but sometimes it takes more than 5 hours of time.

Here is log I have captured from the nodemanager where it got stuck:

34,800 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=user5IP=172.20.178.106OPERATION=Stop Container RequestTARGET=ContainerManageImplRESULT=SUCCESSAPPID=application_1491671269638_43231CONTAINERID=container_1491671269638_43231_01_000005 2017-04-23 01:07:34,800 INFO container.ContainerImpl (ContainerImpl.java:handle(1163)) - Container container_1491671269638_43231_01_000005 transitioned from LOCALIZING to KILLING 2017-04-23 01:07:34,801 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11_tmp 2017-04-23 01:07:34,803 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(492)) - delete returned false for path: [/hd_data/disk21/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/11_tmp] 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk22/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/10 2017-04-23 01:07:34,803 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(489)) - Deleting path : /hd_data/disk22/hadoop/yarn/local/usercache/user5/appcache/application_1491671269638_43231/filecache/10_tmp 2017-04-23 01:07:34,803 WARN hdfs.BlockReaderFactory (BlockReaderFactory.java:getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader. java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:681) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3497) at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:775) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:692)

Do anyone have any experience with this kinds of issue.

Thanks

Khireswar

1 REPLY 1
Highlighted

Re: Hive query taking long time and stuck on a node manager with error getRemoteBlockReaderFromTcp(714)) - I/O error constructing remote block reader

@khireswar Kalita

There are high chances that your connection was not configured properly or it might be getting reset.

You might have to hit on the connection configuration.

Don't have an account?
Coming from Hortonworks? Activate your account here