Reply
New Contributor
Posts: 2
Registered: ‎07-23-2014

LeaseExpiredException and file not found

Hello,

 

I am using 

Hadoop 2.0.0-cdh4.7.0

 

There are several posts about LeaseExpiredException but dint see anything alluding to this specific trace. This seem to be some race condition. Looks like under /user/ctippur/base, the temp files get created and then gets deleted prematurely.

 

Is there any property I can set?

 
 

[DFSClient] - Failed to close file /user/ctippur/base/2014-07-22-07-11-46/_temporary/_attempt_local246555922_0001_m_000000_0/data.DUMMY_LOG.0.1.1406012400000-m-00000

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/ctippur/base/2014-07-22-07-11-46/_temporary/_attempt_local246555922_0001_m_000000_0/data.DUMMY_LOG.0.1.1406012400000-m-00000: File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-256116679_1, pendingcreates: 1]

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2609)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2426)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2339)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:299)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44954)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

 

at org.apache.hadoop.ipc.Client.call(Client.java:1238)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)

at com.sun.proxy.$Proxy9.addBlock(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)

at com.sun.proxy.$Proxy9.addBlock(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:291)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1228)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1081)

at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)

 

I dont see that folder on hdfs.

 

On the Namenode logs ..

 

2014-07-22 00:12:10,228 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ctippur (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/ctippur/base/2014-07-22-07-11-46/_temporary/_attempt_local246555922_0001_m_000000_0/data.DUMMY_LOG.0.1.1406012400000-m-00000: File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-256116679_1, pendingcreates: 1]

2014-07-22 00:12:10,228 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from10.132.63.29:36708: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/ctippur/base/2014-07-22-07-11-46/_temporary/_attempt_local246555922_0001_m_000000_0/data.DUMMY_LOG.0.1.1406012400000-m-00000: File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-256116679_1, pendingcreates: 1]

org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/ctippur/base/2014-07-22-07-11-46/_temporary/_attempt_local246555922_0001_m_000000_0/data.DUMMY_LOG.0.1.1406012400000-m-00000: File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-256116679_1, pendingcreates: 1]

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2609)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2426)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2339)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:299)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44954)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

 

I am using CDH4. Can someone please point to what could be wrong.

- S

New Contributor
Posts: 2
Registered: ‎07-23-2014

Re: LeaseExpiredException and file not found

I see that there is a similar issue ...

http://community.cloudera.com/t5/Batch-Processing-and-Workflow/LeaseExpiredException/m-p/371#M30

 

But I am using a 3rd party solution with includes MR job (https://github.com/linkedin/camus).

I am not sure if this is a code issue or a config issue.

 

- S

Highlighted
New Contributor
Posts: 2
Registered: ‎12-24-2018

Re: LeaseExpiredException and file not found

Did anybody solve thi issue ,settiing spark.speculation=false is not helping either

 

Announcements