Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cannot obtain block length for LocatedBlock

avatar
Contributor

When I do "hdfs dfs -cat file1" from the command line, I got the exception saying that it Cannot obtain block length for LocatedBlock. How should we handle this case?

1 ACCEPTED SOLUTION

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
6 REPLIES 6

avatar
Guru

you can try running "hdfs fsck <path>/file1" to check if there is a corruption. Next step would be to try and recover the blocks or if not relevant, then remove them.

avatar
Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer

"Cannot obtain block length for LocatedBlock" error comes because of file is still in being-written state. Run the fsck command to get more information about the error file.

$ hdfs fsck -blocks /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593219600494.txt.gz

Connecting to namenode \ugi=hdfs&blocks=1&path=%2Fuser%2Fbdas%2Fwarehouse%2Fqueue%2Finput%2Fapollo_log%2Fds%3D20200626%2Fhr%3D21%2FDN27.Apollo.1593219600494.txt.gz

FSCK started by hdfs (auth:KERBEROS_SSL) from /10.40.29.101 for path /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593219600494.txt.gz at Sat Jun 27 08:33:46 EDT 2020

Status: HEALTHY

 Total size:    0 B (Total open files size: 21599 B)

 Total dirs:    0

 Total files:   0

 Total symlinks:                      0 (Files currently being written: 1)

 Total blocks (validated):       0 (Total open file blocks (not validated): 1)

 

We can run fsck for the full directory to check for all the file's status:

~]$ hdfs fsck /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/ -files -openforwrite

Connecting to namenode \ugi=hdfs&files=1&openforwrite=1&path=%2Fuser%2Fbdas%2Fwarehouse%2Fqueue%2Finput%2Fapollo_log%2Fds%3D20200626%2Fhr%3D21

FSCK started by hdfs (auth:KERBEROS_SSL) from /10.47.27.101 for path /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21 at Sat Jun 27 08:47:32 EDT 2020

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21 <dir>

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593219600494.txt.gz 21599 bytes, 1 block(s), OPENFORWRITE:  OK

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593220237244.txt.gz 20661944 bytes, 1 block(s):  OK

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593220269292.txt.gz 20857646 bytes, 1 block(s):  OK

 

In above output it is showing only 1-file is having issue. So run the below command to recover its lease.

$ hdfs debug recoverLease -path /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593219600494.txt.gz -retries 3

 

Once succeeded, we will verfiy again with fsck command:

$ hdfs fsck /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/ -files -openforwrite

Connecting to namenode via ugi=hdfs&files=1&openforwrite=1&path=%2Fuser%2Fbdas%2Fwarehouse%2Fqueue%2Finput%2Fapollo_log%2Fds%3D20200626%2Fhr%3D21

FSCK started by hdfs (auth:KERBEROS_SSL) from /10.40.29.101 for path /user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21 at Sat Jun 27 08:49:09 EDT 2020

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21 <dir>

/user/bdas/warehouse/queue/input/apollo_log/ds=20200626/hr=21/DN27.Apollo.1593219600494.txt.gz 3528409 bytes, 1 block(s):  OK

 

Now the error is resolved.

avatar
New Contributor

thx for solution. 

 

Job works fine

avatar
New Contributor

Hi DarveshK,

do you know what to do if hdfs debug recoverLease returns

recoverLease returned false.
Giving up on recoverLease

 

Does this mean that the file is lost?

 

Thank you!

avatar
Expert Contributor

@diplompils  It is not necessary that file is lost if you are getting the output as false for recoverLease command. Usually file can't be deleted until it has lease acquired and not explicitly deleted using rm command. 

 

You can try below-

hdfs debug recoverLease -path <file> -retries 10

Or you may check - https://issues.apache.org/jira/browse/HDFS-8576