Support Questions
Find answers, ask questions, and share your expertise

Cannot free lease if client terminates with "all datanodes are bad. Aborting..."

Highlighted

Cannot free lease if client terminates with "all datanodes are bad. Aborting..."

Explorer

I have a single node HDFS system. Occasionally, when appending to a file, I get an error that the client had trouble reading from the server:

Exception in createBlockOutputStream blk_1073741908_2968: java.io.EOFException: Unexpected EOF while trying to read response from server

Then, when I try to close the FSDataOutputStream, the client reports all datanodes are bad:

All datanodes [DatanodeInfoWithStorage[192.168.1.211:50010,DS-d234bb2c-7498-4050-91af-4bf81a758446,DISK]] are bad. Aborting...

However, the datanode is still functional and has not crashed. When I try to append to the same file again, the client reports the error:

Failed to APPEND_FILE /PSG/VideoVault/VideoStorage/112/2020061910.idx for DFSClient_NONMAPREDUCE_1526892566_159 on 192.168.1.211 because DFSClient_NONMAPREDUCE_1526892566_159 is already the current lease holder.

Though the client believes that it has severed the connection to the file, the namenode still seems to believe that it is being written and will no longer allow me to append to the file.

 

Any idea what is going wrong here or how to clear my lease out of the NameNode so I can once again try and append to the file?

 

Thanks, David