Hi TS, are you still facing this issue too? Have you changed back to 3 replicates? Or still configured with 2?   1) Should I se "hadoop fs -setrep" to change the replication factor of certain files? JMS: No. Keep it the way it is for now.   2) What's the manual way to 'force' the affected blocks to replicate themselves? JMS: It depends... If they are configure to replicate 100 times, you might not have enought nodes and you can not force that. How many nodes do you have in your cluster? Car you past here part of the fsck output?   3) Should I remove permanetly certain types of files?     For instance, in the fsch log report I am seeing a lot of files with of this type: /user/hue/.Trash/150507010000/user/hue/.cloudera_manager_hive_metastore_canary/hive0_hms/cm_test_table1430446320640/p1=p1/p2=421 <dir> /user/hue/.Trash/150507010000/user/hue/.cloudera_manager_hive_metastore_canary/hive0_hms/cm_test_table1430446620772 <dir> /user/hue/.Trash/150507010000/user/hue/.cloudera_manager_hive_metastore_canary/hive0_hms/cm_test_table1430446620772/p1=p0 <dir> JMS: This is the trash. If you don't need those file, clean the trash?   4) How about the /tmp/logs/ files? Dp I reset their setrep setting or periodically remove them? JMS: Same thing. Temporary files. Can you list them to make sure? You might be able to delete them.   5) I am also having quite a few Accumulo tables reporting under-replicated blocks!  JMS: Here again, please paste the logs here. This one is the most concerning. They should have the default, except if accumulo set that to more than the factor 2 you have.   JMS ... View more

So ZK is working fine. Perfect.   Make sure HBase is down and clear everything: hadoop fs -rm -r /hbase/* echo "rmr /hbase" | zookeeper-client   Then try to start HBase.   JMS ... View more

Ok. so here is the complete situation. When you run a MR on top of a Snapshot, the MR framework will look at all the inputs and create all the tasks for that. However, those tasks might have to wait for some time to be executed depending on the number of slots available on the cluster vs the number of tasks. The issue is, if while the tasks are pending one of the input is move/deleted/split/merged, etc. then the splits are not pointing anymore to a valid input and the MR job wil fail. To avoid that, we have to create al lthe links to all the inputs to make sure HBase keep a reference to those files even if they have to me moved, the same way a snapshot is doing. The issue is, those links have to be in the /hbase folder. And this is why you need the rights for that. So to be able to run a MR job on top of a snapshot you need a user with reads/writes access to the /hbase folder. This should be fixed in HBase 1.2 (but it's just on the plans for now and you will need to double check wen we will be closer to 1.2). Also, please keep in mind that doing MR on top of Snapshots bypass all the HBase layers. Therefore, if there is any ACLs or Cell level security activated on the initial table, then will all by bypassed by the MR job. Everything will be readable by the job. Let me know if you have any other question or if I can help with anything. HTH. JMS ... View more