Findings:- In Cloudera Manager, the Block Count Threshold warnings are displayed.   Root Cause:- Check the following: Go to   Cloudera Manager   >   HDFS   >   Configuration   >   DataNode Block Count Thresholds Go to   Cloudera Manager   >   HDFS   >   WebUI   >   NameNode Web UI   > click on   DataNode   tab > Get the block count of your node If the block count of your node is greater than DataNode Block Count Thresholds, the Block Count warning is displayed. Also, "presence of many small files" also creates this warning as the DataNode will have too many small blocks in number, which hit the threshold value.   Resolution:- To resolve this issue, follow any of these instructions: Increase the   DataNode Block Count Thresholds   in Cloudera Manager at 'a' above. Clean up unwanted data, but if your   Trash   folder maintains old data (for example, 24 hours) then you will see the result after 24 hours only Add additional DataNodes and apply rebalance Have sufficient heap assigned to the DataNode process. A good rule of thumb is that 1M blocks on a DataNode need at least 1 GB heap assigned to the process. You can also ignore this warning, but just keep an eye on block pool usage percentage from 'b' above.   ... View more

Methods to migrate Phoenix table from one environment to other Use Case 1 - Migrating table data without INDEX. As all Phoenix data is present as HBase tables. Therefore, the easiest way is to take a snapshot of all the tables in HBase, and then use HBase to copy them over Refer to Hbase table copy from one cluster to other Once the steps in the above article are complete, create Phoenix view on existing HBase table in the destination cluster Refer to   How I map Phoenix table to an existing HBase table? Use Case 2 - Migrating table data with INDEX. To migrate all Phoenix table data with INDEXING from source to target, follow above step and then recreate INDEX in target cluster. Please refer to https://phoenix.apache.org/secondary_indexing.html   Note: 1. There is no way to export the schema as DDL statements from Phoenix currently; it is in the pipeline 2. Manually editing the   SYSTEM.CATALOG   is not recommended as it may cause other discrepancy ... View more

This article describes -- 1. Why taking backup of HBase dataset using HBase Backup & Restore utility cause log cleaner to not remove the logs. Eventually high HDFS utilization. 2. How to prevent/get rid of this situation.   Findings:-  When using   HBase Backup & Restore   utility to take backup of HBase dataset, HBase backup log cleaner is unable to remove the log and keeps it in oldWALs, increasing its directory size inside HDFS. The following exception is displayed in HBase master log: master.BackupLogCleaner: Didn't find this log in hbase:backup, keeping: hdfs://<NAMESERVICE>/apps/hbase/data/oldWALs/<XXXXXX>%2C16020%2C1597226939896.default .1597234319261   Root Cause:- This issue occurs because HBase is designed in a way that it accumulates the logs in oldWALs after creating an HBase backup. This is done to avoid any data missing in backup. After the first backup sessions WAL files get accumulated in old WAL directory until the next backup session.   Ref- https://issues.apache.org/jira/browse/HBASE-14123   Resolution:- To resolve this issue, either run backups regularly along with old backup removal, or disable backups in the system. HBase Snapshot can be another good alternative to avoid such behaviour for HBase data backup.   Note:- For more information on HBase Backup Utility, refer to the following links: Backup and Restore Backing up and restoring Apache HBase datasets Command for creating HBase backup image ... View more

This article describes why SBNN crashes with an OOM error when audit to Solr is enabled in the cluster, and the way to resolve this issue.   Findings:- Active NameNode gets crashed and becomes Standby Name Node (SBNN), and further fails to start with no further logs updated in .log. The following traces are found from the .out file: # java.lang.OutOfMemoryError: Requested array size exceeds VM limit # -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"   Root Cause:- 1. The issue here is, if the audit to Solr does not work, it will cause the Spool folder to be filled up and Namenode will become busy. If the audit framework detects that an audit destination is down, it buffers the audit messages in memory. Once the memory buffer fills up, it can be configured to spool the unsent messages to disk files [Namenode log dir] to prevent or minimize the loss of audit messages. We can see frequent   "Destination is Down"   traces in the active NameNode log file in case the audit to Solr does not work.   2. Please note if the audit to Solr does not work, this may further fill Active NN log dir also and crash Active NameNode.    > If this issue persists it may cause fast filling of Active NameNode log dir also and further crash the Active NN also due to "100% utilisation of log directory mount point".   Resolution:- To bring back SBNN, move all its logs and audit dir into backup sub-directory at  /var/log/hadoop/hdfs/old_logs . Also, fix the audit to Solr issue. ... View more

There are several options available to achieve this use case.   The easiest and the best approach would be HBase snapshots method to transfer the data.     Note: All actions need to be performed as the HBase user only, to ensure correct permissions. On source cluster: #hbase shell> snapshot ‘Test’_Table’, ‘Test_Table_SS’ #hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats #hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot 'Test_Table_SS' -copy-to hdfs://<Destination_NN_hostname>:8020/hbase -mappers 16 -bandwidth 200 On destination cluster: #hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -snapshot ‘Test_Table_SS’ -files -stats #hbase shell> clone_snapshot ‘Test_Table_SS’, ‘Test_Table’ #hbase shell> major_compact 'Test_Table' Once done, you can choose to delete the snapshots on both ms05 and as05: #hbase shell> delete_snapshot 'Test_Table_SS'​ But If you plan to use CopyTable , this will not work without additional configuration. Communication between an older client and a newer server is not guaranteed, there is currently a workaround to allow this to work by adding the following property to your client configuration: On the client being used to launch the CopyTable, you can do either: Command Line: -Dhbase.meta.replicas.use=true OR       hbase-site.xml: hbase.meta.replicas.use true       ... View more