So i shouldn't search for the missing 40 T and the right storage is what the fsck shows? ... View more

CM 5.5.2 CDH 5.5.4 ... View more

Yes ... View more

Hi Lars,   In your comments before you asked me to check the UID and i mentioned the version 'm using, unfortunately, it doesn't work. ... View more

Do you think looking at the edit logs size when this occur should be a good indication? ... View more

Hi,   When i Run fsck on my cluster i got that several blocks under replicated and the target replication is 3 even i changed the dfs.replication to NN/ DNs and client server to replication factor 2, and mapred.submit.replication changed to 2. tried also: <property> <name>dfs.replication</name> <value>2</value> <final>true</final> </property> I also restarted all service at my cluster including the oozie. Looking at one of the running jobs conf and see the following with replication factor 3: mapreduce.client.submit.file.replication s3.replication kfs.replication dfs.namenode.replication.interval ftp.replication s3native.replication   ... View more

The intersting thing that i noticed when this happened at the same time some jobs that runs once a day write to HDFS relatively too much data and it's run with a good number of reducers betweeb 400-1100, which make me suspect in the blocks that written by these jobs at the same time and the vm is getting some lag, trying to find a way to approve this. ... View more

Total size: 253714473531851 B (Total open files size: 11409372739 B) Total dirs: 1028908 Total files: 7639121 Total symlinks: 0 (Files currently being written: 107) Total blocks (validated): 8781147 (avg. block size 28893090 B) (Total open file blocks (not validated): 149) Minimally replicated blocks: 8781147 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.8528664 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 30 Number of racks: 1 FSCK ended at Mon Feb 20 21:33:23 EST 2017 in 190136 milliseconds The filesystem under path '/' is HEALTHY hadoop fs -du -s / 244412682417174 708603392967605 / hadoop fs -count -q / 9223372036854775807 9223372036846392726 none inf 987886 7395195 244417466380498 /   the non HDFS reserved space is 10 GB and for 30 nodes so it's should not exceed 1 T with replication factor 3.   it's really annoying.     ... View more