Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

insert overwrite cause hdfs error: Unknown error 255

Highlighted

insert overwrite cause hdfs error: Unknown error 255

Explorer

The following query is getting hdfs error:

insert overwrite bigdata.nodestats_swap partition (year,month,day,hour)  select * from bigdata.nodestats where year=2016 and month=4 and day=5 and hour=11

 

It does not happen very often but would like to understand why\what happens.

Below are the imala and namenode logs. 

Thanks in advance.

 

Impala log:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3602)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3399)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3255)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:667)

        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

 

        at org.apache.hadoop.ipc.Client.call(Client.java:1468)

        at org.apache.hadoop.ipc.Client.call(Client.java:1399)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

        at com.sun.proxy.$Proxy14.addBlock(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)

        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy15.addBlock(Unknown Source)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)

        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)

I0405 06:24:37.250735 19067 runtime-state.cc:230] Error from query 724a04f162c946f2:a85c6f0bcd88e99f: Failed to close HDFS file: hdfs://nameservice1/user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f//.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq

Error(255): Unknown error 255

 

 

Relevant Namenode logs:

2016-04-05 06:23:39,449 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.2.parq. BP-1123871397-10.216.32.10-1433806145781 blk_1190477181_116736497{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-caeb90ae-cc3d-4142-9ac1-d83290ba9d4a:NORMAL:10.216.32.105:50010|RBW], ReplicaUnderConstruction[[DISK]DS-2a57f3a9-b316-481f-a3a8-d2430f07dba5:NORMAL:10.216.34.101:50010|RBW], ReplicaUnderConstruction[[DISK]DS-f90cdf95-8c43-4e81-ad37-d2c0dc384634:NORMAL:10.216.34.105:50010|RBW]]}

 

2016-04-05 06:23:40,770 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.2.parq is closed by DFSClient_NONMAPREDUCE_-569971520_1

 

2016-04-05 06:24:37,163 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:impala (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.

2016-04-05 06:24:37,163 INFO org.apache.hadoop.ipc.Server: IPC Server handler 249 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.216.32.105:43028 Call#694292838 Retry#0: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.

 

Don't have an account?
Coming from Hortonworks? Activate your account here