question Re: Unable to load into HBase due to error with data blocks in Archives of Support Questions (Read Only)

Unable to load into HBase due to error with data blocks

joshua_adeleke — Fri, 08 Jul 2016 22:44:48 GMT

I get an error "Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 18, <datanode>): java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1426797840-1461158403571:blk_1089439824_15699635; getBlockSize()=0; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[,DISK], DatanodeInfoWithStorage[,DISK], DatanodeInfoWithStorage[,DISK]]}"

The background to this is: We changed our Oozie DB to MySQL and on restarting the cluster, the namenode failed with connectionRefused error. It was started manually from the CLI and after restarted with Ambari and it worked fine. I have used hdfs fsck to check for corrupt files but i get a 'healthy' status report.

Any clue as to how i can get past this issue? @Kuldeep Kulkarni @Artem Ervits @Sagar Shimpi @Benjamin Leonhardi

Re: Unable to load into HBase due to error with data blocks

srai1 — Fri, 08 Jul 2016 22:47:36 GMT

https://community.hortonworks.com/questions/37412/cannot-obtain-block-length-for-locatedblock.html

Re: Unable to load into HBase due to error with data blocks

elserj — Fri, 08 Jul 2016 22:50:27 GMT

Can you share the hdfs fsck command you ran? It definitely sounds like HDFS is not healthy.

Re: Unable to load into HBase due to error with data blocks

joshua_adeleke — Fri, 08 Jul 2016 23:00:00 GMT

@srai Please find below the report i got from running hdfs fsck /

...............................Status: HEALTHY Total size: 84086783290897 B (Total open files size: 35725218 B) Total dirs: 255918 Total files: 7090531 Total symlinks: 0 (Files currently being written: 464) Total blocks (validated): 7131287 (avg. block size 11791249 B) (Total open file blocks (not validated): 86) Corrupt blocks: 0 Number of data-nodes: 8 Number of racks: 1 FSCK ended at Fri Jul 08 16:43:47 SAST 2016 in 141368 milliseconds The filesystem under path '/' is HEALTHY

Re: Unable to load into HBase due to error with data blocks

joshua_adeleke — Fri, 08 Jul 2016 23:37:24 GMT

Here is another one below, Josh.

Status: HEALTHY Total size: 84184775260004 B (Total open files size: 36288883 B) Total dirs: 255954 Total files: 7102482 Total symlinks: 0 (Files currently being written: 456) Total blocks (validated): 7143238 (avg. block size 11785240 B) (Total open file blocks (not validated): 79) Minimally replicated blocks: 7143238 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 130 (0.0018199029 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.9979758 Corrupt blocks: 0 Missing replicas: 257 (0.0012000647 %) Number of data-nodes: 8 Number of racks: 1 FSCK ended at Fri Jul 08 18:29:19 SAST 2016 in 239594 milliseconds The filesystem under path '/' is HEALTHY

Re: Unable to load into HBase due to error with data blocks

elserj — Sat, 09 Jul 2016 00:24:04 GMT

Thanks, @Joshua Adeleke. Like in the other question linked by Srai, if you know the specific file(s) your job is reading, you could try to use the `hdfs debug recoverLease` command on those files. Normally, a lease on an HDFS file will expire automatically if the writer abnormally goes away without closing the file. If you are sure no client is trying to write the file, you could try the recoverLease to force the NN to let this operation succeed.

Re: Unable to load into HBase due to error with data blocks

joshua_adeleke — Mon, 01 Aug 2016 14:09:43 GMT

This issue was resolved by restarting the namenode.