Reply
Highlighted
Explorer
Posts: 18
Registered: ‎10-25-2016

CDH5.10 upgrade from CDH5.8.3 deleted hdfs dirs, files

[ Edited ]

I upgraded CDH from CDH5.8.3 to CDH5.10.2 last month on one of my cluster.
We started noticing job failures. These were due to some partitions of an external table in hive missing on hdfs but not on hive.

Hive metadata had those partitions but somehow were missing from hdfs.

We are not sure if the upgrade deleted those partitions or someone else did.

But looking at below fsck logs it looks like there were missing files, dirs after upgrade.

Below are the fsck logs of before and after upgrade of CM and CDH.
Any one has any thoughts on why an upgrade of CM and CDH would delete some files on hdfs?

This has happened for my other cluster too. Not a lot of critical files were deleted, so it went unnoticed.
Note: No job ran or deleted any files manually before and after upgrade when below fsck was ran. Cluster was down during upgrade.


Pre upgrade

[root@namenode ~]# sudo -u hdfs hdfs fsck /
Connecting to namenode via http://namenode.hadoop.com:50070
FSCK started by hdfs (auth:SIMPLE) from /10.124.15.130 for path / at Wed Nov 01 13:11:16 EDT 2017.
.
........Status: HEALTHY
Total size: 8929429582847 B (Total open files size: 3758101818 B)
Total dirs: 473886
Total files: 4302408
Total symlinks: 0 (Files currently being written: 4)
Total blocks (validated): 3931888 (avg. block size 2271028 B) (Total open file blocks (not validated): 32)
Minimally replicated blocks: 3931888 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9979231
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 15
Number of racks: 1
FSCK ended at Wed Nov 01 13:13:48 EDT 2017 in 96225 milliseconds
The filesystem under path '/' is HEALTHY


Post upgrade
[root@namenode ~]# sudo -u hdfs hdfs fsck /
Connecting to namenode via http://namenode.hadoop.com:50070
FSCK started by hdfs (auth:SIMPLE) from /10.124.15.130 for path / at Wed Nov 01 13:11:16 EDT 2017
.
.
..................................................................................Status: HEALTHY
Total size: 8957018013796 B (Total open files size: 3758101818 B)
Total dirs: 472149
Total files: 4291382
Total symlinks: 0 (Files currently being written: 5)
Total blocks (validated): 3920904 (avg. block size 2284426 B) (Total open file blocks (not validated): 32)
Minimally replicated blocks: 3920904 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 2.9979224
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 15
Number of racks: 1
FSCK ended at Wed Nov 01 17:17:33 EDT 2017 in 83829 milliseconds


The filesystem under path '/' is HEALTHY

Abhishek
Announcements