Hi, I'd like to share a situation we encountered where 99% of our HDFS blocks were reported missing and we were able to recover them. We had a system with 2 namenodes with high availability enabled. For some reason, under the data folders of the datanodes, i.e /data0x/hadoop/hdfs/data/current - we had 2 Block Pools folders listed (example of such folder is BP-1722964902-1.10.237.104-1541520732855). There was one folder containing the IP of namenode1 and another containing the IP of namenode 2. All the data was under the BlockPool of namenode 1, but inside the VERSION files of the namenodes (/data0x/hadoop/hdfs/namenode/current/) the BlockPool id and the namespace ID were of namenode 2 - the namenode was looking for blocks in the wrong block pool folder. I don't know how we got to the point of having 2 block pools folders, but we did. In order to fix the problem - and get HDFS healthy again - we just needed to update the VERSION file on all the namenode disks (on both NN machines) and on all the journal node disks (on all JN machines), to point to Namenode 1. We then restarted HDFS and made sure all the blocks are reported and there's no more missing blocks. ... View more

@Artem Ervits, thanks for sharing this link. ... View more

@Karthik Gopal, thanks for sharing this link. ... View more

You can check this video blog for step by step process https://www.youtube.com/watch?v=vB1SN0LBicE , You can have look at this video blog on Ambari LDAP Integration https://www.youtube.com/watch?v=vB1SN0LBicE ... View more

Hi @Rushikesh Deshmukh The following table provides an overview for quickly comparing these approaches, which I’ll describe in detail below. http://blog.cloudera.com/blog/2013/11/approaches-to-backup-and-disaster-recovery-in-hbase/ i used distcp as well but that did not work for me , in the sense data was copied but while running hbck i had issue if you want to create backup on same cluster then copytable and sanpshot are very easy for inter cluster snapshot works good let me know if you need more details Also this below link is really very useful and clear http://hbase.apache.org/0.94/book/ops.backup.html ... View more

@Artem Ervits, I was referring for both hdfs and hbase, got required answer. But, thanks for your suggestion. ... View more

@Benjamin Leonhardi, thanks for sharing this useful information and link. ... View more

Just wanted to point out that Hazelcast is an in-memory data grid not a data store. With standard configuration, the data is entirely in volatile memory. You can use a backing store to ensure that data is persisted between restarts but the purpose of Hazelcast and IMDGs in general is for application acceleration not data storage. IMDGs are also capable of recieveing and distributing instruction sets across the cluster (send compute to data) similar to Hadoop. IMDGs can also execute instructions on every individual get/put/delete operation that hits the cluster. At the moment, IMDGs are not designed to scale past several TB and so would generally be used to augment a big data architecture, not replace it. However, the potential acceleration provided by and IMDG to an OLTP use case can be in the n^x realm. ... View more

@Artem Ervits, thanks for sharing this information. ... View more