Your region server size is just about right and memstore flush size seems reasonable too. How many column families do you have in your tables? Use the following to determine your number of regions Usage of the region server's memstore largely determines the maximum number of regions for the region server. Each region has its own memstores, one for each column family, which grow to a configurable size, usually between 128 and 256 Mb. Administrators specify this size with the hbase.hregion.memstore.flush.size property in the hbase-site.xml configuration file. The region server dedicates some fraction of total memory to region memstores based on the value of the hbase.regionserver.global.memstore.size configuration property. If usage exceeds this configurable size, HBase may become unresponsive or compaction storms might occur. Use the following formula to estimate the number of Regions for a region server: (regionserver_memory_size) * (memstore_fraction) / ((memstore_size) * (num_column_families)) For example, assume the following configuration: region server with 16 Gb RAM (or 16384 Mb) Memstore fraction of .4 Memstore with 128 Mb RAM 1 column family in table The formula for this configuration would look as follows: (16384 Mb * .4) / ((128 Mb * 1) = approximately 51 regions The easiest way to decrease the number of regions for this example region server is increase the RAM of the memstore to 256 Mb. The reconfigured region server would then have approximately 25 regions, and the HBase cluster will run more smoothly if the reconfiguration is applied to all region servers in the cluster. The formula can be used for multiple tables with the same configuration by using the total number of column families in all the tables. ... View more

According to following link, this is the expected behavior (notice, show database is not mapped to Ranger) https://cwiki.apache.org/confluence/display/RANGER/Hive+Commands+to+Ranger+Permission+Mapping ... View more

Snapshots capture metadata at point in time so you can recover the state of your cluster to certain point in time if someone accidentally deletes something or due to some other failure if you would like to go back to certain point in time. One thing to remember with snapshots is that they only capture state at a point in time. No data copying occurs with Snapshot. It's a pretty quick O(1) operation. However, if you are creating many snapshots, then when you delete data, Hadoop will see a snapshot pointing to that data and instead of deleting your data, it will move it to an archiving folder. Many people are surprised that somehow their disk space is not freeing up even though they have deleted a lot of data (they don't even see deleted files as they have been moved to an archiving folder). The culprit in these cases is usually Snapshot. But, see this is barely enough to make true backups. Across the industry, the mechanism for backups and disaster recovery is replication (both for Hive and HDFS as well as for HBase). So what you should be looking at is replication and then see if making snapshots also makes sense for your use cases (its almost always useful to have some snapshots - fewer snapshots means less disk space use). Tp copy data between clusters, you will use distcp. Check the following link: https://hadoop.apache.org/docs/r1.2.1/distcp2.html If my answer helped, please accept. ... View more

But you still have 6200 regions per region server? Is that right? That itself can cause a lot of master activity. You need to change your settings so you have less regions per region server. What is the heap size of your region server? What is memstore size for each table (hbase.hregion.memstore.flush.size)? ... View more

@Naresh Kumar Korvi Can you update your minimum number of entries to what you want at minimum, let's say 500 files. Also since it's all similar data going into one file, I am assuming flow file attributes are same. Can you change your merge strategy to defragment? Finally I am your flow is something like this: consumeKafka -> mergecontent -> putHDFS Is that right? ... View more

@hardik desai Just to add to Artem's answer, yes what you are trying to do is possible with HDFS federation but like Artem said, it's not supported. What is your motivation of doing it? Do you really have multiple PBs of data where you will have 1000's of nodes and your namenode wil not be able to handle the cluster? If not, then federation is not the right way to implement multi tenancy. ... View more

I think I understand your point. See my new answer. ... View more