question Re: corrupted block issue..i have 100+ corrupted blocks in Archives of Support Questions (Read Only)

corrupted block issue..i have 100+ corrupted blocks

g_rao9704 — Fri, 05 Aug 2016 18:37:15 GMT

I have 4 nodes cluster with NN-HA ..when i restart two datanodes i have 100+ corrupted blocks(shown in ambari>hdfs>servieces ) and i have replication factor 3 so i want to deleted these corrupted blocks..before i deleted this i want to check which are files are corrupted so i run below command and got below ouput ,here did not get any corrupted blocks or files? please help me ..i have this issue since long ..

P1-230-8:/root> hdfs fsck / | egrep -v '^\.+

Connecting to namenode via http://stlpr8711:50070 FSCK started by hdfs (auth:SIMPLE) from /39.6.64.8 for path / at Fri Aug 05 06:27:52 CDT 2016

...........................Status: HEALTHY T

otal size: 39189208876540 B (Total open files size: 83162 B)

Total dirs: 710383 Total files: 2184527

Total symlinks: 0 (Files currently being written: 7) Total blocks (validated): 2309173 (avg. block size 16971101 B) (Total open file blocks (not validated): 7)

Corrupt blocks: 0

Number of data-nodes: 4

Number of racks: 1

FSCK ended at Fri Aug 05 06:28:40 CDT 2016 in 47634 milliseconds The filesystem under path '/' is HEALTHY

Re: corrupted block issue..i have 100+ corrupted blocks

vbonthu — Fri, 05 Aug 2016 18:38:38 GMT

@sankar rao

In the above hdfs fsck command output I see corrupted blocks 0. Generally after you restart the nodes it takes some time for namenode to figure out with blocks location and health state.

Re: corrupted block issue..i have 100+ corrupted blocks

g_rao9704 — Fri, 05 Aug 2016 18:48:05 GMT

@Vinod Bonthu

Thanks Vinod..but restart happens 20 days back and still in ambari shows that critical warning and 109 corrupted blocks.....

Re: corrupted block issue..i have 100+ corrupted blocks

jyadav — Fri, 05 Aug 2016 18:51:09 GMT

Hi @sankar rao

This seems to be an known issue with Ambari, please refer below bug for more info and see if version matches.

https://issues.apache.org/jira/browse/AMBARI-12931

Re: corrupted block issue..i have 100+ corrupted blocks

g_rao9704 — Fri, 05 Aug 2016 19:04:18 GMT

@Jitendra Yadav

Thank you for quick response jitendra..i could not see any version matches..i am using hdp 2.1 with ambari 1.6 can you suggest any other alternative solution for this issue...?

Re: corrupted block issue..i have 100+ corrupted blocks

jyadav — Fri, 05 Aug 2016 20:48:40 GMT

I would highly suggest you to upgrade your ambari server with latest release. As a workaround restart the Ambari (server and agents) as well as Ambari Metrics and see if that resolve the issue.

Re: corrupted block issue..i have 100+ corrupted blocks

vbonthu — Fri, 05 Aug 2016 20:59:16 GMT

@sankar rao Restarting the Ambari server as work around.

Re: corrupted block issue..i have 100+ corrupted blocks

g_rao9704 — Sat, 06 Aug 2016 02:18:15 GMT

@Jitendra Yadav

Thanks you .. i will let you the result after restart ambari server..