I can reproduce this easily with some newer versions' shells. The problem seems to be that the newer versions supply a default scanner caching value of Integer.MAX_VALUE, which the older servers didn't have a checking guard against. When they receive such a numeric request, they try to construct a normal ArrayList of that arbitrary size in-memory which Java cannot do as the size exceeds the maximum allowed array size of a JVM. It therefore throws an OOME which combined with the configuration flag of 'Kill when OutOfMemory' destroys the RS with a kill -9 along with an OOME message in the RS's stdout file. The enhancements of HBASE-11544 rearchitected around this by adding guards and other better ways of handling scanner requests. So if your server runs any release > CDH 5.5.0 which was the earliest to include this JIRA, it will not crash even if such a request came to it. In your current situation, aside of a CDH upgrade, I think you can guard against a cascading automatic kill -9 from a rogue client request by disabling the check inside your CM -> HBase -> Configuration called "Kill When Out of Memory" for the RegionServers group. Save and restart all RegionServers after making this change. If any fatal OOMEs occur in important threads within the RS it is already designed to go down, but a client-request OOME can be safely ignored with this flag unset. Once you upgrade, you can revert it back to the default value of enabled, as the issue wouldn't occur on newer HBase versions. Does this help? ... View more

The configs you are targeting are configs local to each NodeManager. You can configure each NodeManager distinctly and still have a working (heterogenous) cluster. The RM views the cluster as an aggregate but does consider each NM's resource limits as its own published value when they register. If you are using Cloudera Manager to install and manage your cluster, then you are looking for the concept of 'Role Groups' as explained here: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_role_groups.html. The role groups layer lets you divide your set of hosts and their roles into groups of differing configurations, which in your case you can use to divide in two and configure each group's resource.memory.mb separately (into 60 GB and 120 GB as appropriate). If you do not use Cloudera Manager, then I highly recommend it, but you can also manage the NodeManager's local yarn-site.xml of each NodeManager separately -- i.e. have two distinct copies, the one for 60 GB placed only on the older hosts, and the other for 120 GB placed only on the newer hosts. ... View more

> 1) We are also running HBase in this cluster, with RegionServers colocated with the Datanodes. Are there any gotchas that should be considered given this? The DN writes the blocks in typically round-robin'd manner across the disk list, but in your configuration if a disk is found to match the threshold it will select that disk over and over until the threshold gets reached. If the threshold is very large (many thousand full blocks required to reach the divide) then the RS performance can likely suffer a bit when its trying to flush, replay or compact in parallel. This would go away when the disk rotation falls back to round robin due to no volume being in violation of the space threshold. If you're OK in observing some small slowness (assuming a small difference threshold, and disks not being cleaned and reinserted too often) then you should be good. If however your HBase usage is very latency bound, then consider using a smaller preference fraction so it does not focus on pumping all work onto a single or specific set of disks when the threshold is found to be crossed. > 2) We don't have CM Enterprise, so the official rolling restart is not available to us. Is it possible to restart the Datanode role on nodes individually? You can do the restarts one by one from the HDFS -> Instances page or the API, but you'll need to manually ensure that a DN has come back up in a functional, connected state before moving onto another (by checking the DN's metrics or its logs). The enterprise rolling restart does that check automatically as it progresses. CM API is documented at http://cloudera.github.io/cm_api/ ... View more

> 1. Is the Hbase snapshot take only pointer of table or it make copy of entire hbase table. Within the cluster the snapshot is only an increase in metadata ('pointer' as you mention) where the HFiles carrying the data are not copied but are moved to a special location where they are kept for the life of the snapshot existing. > 2. What export snapshot does in hbase. Exporting a snapshot allows you to make an actual, full-data physical copy of the data to another storage/HBase cluster. > 3. Is hbase snapshot allow in cloudera express version. Snapshots by themselves are a feature offered by Apache HBase itself. It comes with your 100% Open Source CDH and is always available for use regardless of your Cloudera Manager version/edition. This would cover all the Snapshot information presented at http://archive.cloudera.com/cdh5/cdh/5/hbase/book.html#ops.snapshots With Cloudera Manager Enterprise, however, you gain the additional abilities to schedule HBase Snapshots and schedule exports of these Snapshots to S3-compliant storages as described at http://www.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_managing_hbase_snapshots.html For (1) and (2) I'd recommend reading also (in-full) the older but still relevant blog post of https://blog.cloudera.com/blog/2013/03/introduction-to-apache-hbase-snapshots/ ... View more

While it may appear possible to do this I'd strongly recommend against it because when you'd read back a written 150 MB MOB cell, it'd give you heap utilisation problems during the RPC encoding and transfer done by the RS. Its probably better to store the larger-than-10 MB files as HDFS files and store their paths in HBase. ... View more

You can find what operations are supported in the hdfs-fuse source: https://github.com/cloudera/hadoop-common/tree/cdh5.8.0-release/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs (this is a directory-tree, look for the files with "impls" in their names for the syscalls supported). Git likely needs more advanced features from its used filesystem that HDFS currently does not offer. You can run git under strace to find out what that syscall is. ... View more

Cloudera offers Backup and Disaster Recovery (BDR) features as part of its enterprise offering that can do HDFS replication to other clusters, Hive metadata and data replication to other clusters, and also HBase snapshot backups to S3. This is documented in detail at https://www.cloudera.com/documentation/enterprise/latest/topics/cm_bdr_about.html Outside of this you can try to use DistCp for HDFS replication but for Hive replication you will need to manually propagate DDL-associated metadata. ... View more