Hello @Masood, I believe you are asking the commands to run in order to determine the active NN apart from CM UI ( CM > HDFS > Instance > NameNode)   From CLI you have to run couple of commands to detemrine the Active/Standby NN   List the namenode hostnames # hdfs getconf -namenodes c2301-node2.coelab.cloudera.com c2301-node3.coelab.cloudera.com   Get nameservice name # hdfs getconf -confKey dfs.nameservices nameservice1   Get active and standby namenodes # hdfs getconf -confKey dfs.ha.namenodes.nameservice1 namenode11,namenode20   # su - hdfs $ hdfs haadmin -getServiceState namenode11 active $ hdfs haadmin -getServiceState namenode20 standby   Get active and standby namenode hostnames   $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode11 c2301-node2.coelab.cloudera.com:8020   $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode20 c2301-node3.coelab.cloudera.com:8020   If you want to get the active namenode hostname from hdfs-site.xml file, you can go through following python script in github –  https://github.com/grakala/getActiveNN .   Thank you  ... View more

Hello @AlexP    Ref: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep   Referring to HDFS document, answers to your questions are inline. [Q1.] How to estimate how much time would this command take for a single directory (without -w)? [A1.] It depends upon the numbr of files in the directory. If you are running setrep against a path which is a directory, then the command recursively changes the replication factor of all files under the directory tree rooted at path. The time varies dependsing on the file count under the path/directory.    [Q2.] Will it trigger a replication job even if I don't use the '-w' flag? [A2.] Yes, replication will trigger without -w flag. However, it is good practice to use -w to ensure all files are having required replication factor set prior to command exits. Please note, the -w flag requests that the command wait for the replication to complete. Though use of -w potentially takes a long time to complete the command but it gurantees the replication factor changed to the specified value.    [Q3.] If yes, does it mean that the NameNode will actually start deleting 'over-replicated' blocks of all existing files under a particular directory? [A3.] Yes, your understanding is correct. The additonal 1 replica of the block will mark the block as over-replicated and same will be deleted from cluster.  This action will be performed for each files under the directory path keeping only 2 replicas of the file blocks.    Hope this helps.      ... View more

Hello @vincentD Please review the stdout and stderr of the DN which going down frequently. You can navigate to CM > HDFS > Instance > Select the DN which went down > Processes > click on stdout/stderr atthe bottom of the page.     I am asking to verify stdout/stderr suspecting an OOM error (due to java heap running out of memory) leading to the DN exit/shutdown abruptly.     If the DN exit is due to OOM Error, please increase the DN heap size to adequate value to get rid off teh issue further. DN heap sizing rule of thumb says: 1 GB heap memory for 1Million blocks. You can verify your block counts on each DN by navigating to CM > HDFS > NN Web UI > Active NN > DataNode and you can see the DN stats on that page showing block counts and disk usage etc.. ... View more

Hello @tuk you can read about Apache HBase  (hbase-operator-tool) HBCK2 Tool here - https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2    This page suggests how to download the hbase-operator-tool which contains the "hbase-hbck2" jar used to fix the inconsistencies in CDH 6.x (hbase 2.x).  hbase-hbck2-1.0.0-SNAPSHOT.jar See the section talks about fixing problems using hbck2 - https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#fixing-problems  ... View more

Hello @tuk , you can find the hbck2 tool for CDH 6.3.3 from Cloudera via a support ticket. Else you can obtain the hbck2 tool available under the HBase distribution directory. See the   HBASE Downloads Page. - https://downloads.apache.org/hbase/hbase-operator-tools-1.0.0/hbase-operator-tools-1.0.0-bin.tar.gz - http://apachemirror.wuchna.com/hbase/hbase-operator-tools-1.0.0/hbase-operator-tools-1.0.0-bin.tar.gz - https://mirrors.estointernet.in/apache/hbase/hbase-operator-tools-1.0.0/hbase-operator-tools-1.0.0-bin.tar.gz   By default, running   bin/hbase hbck, the built-in   hbck1   tooling will be run. To run   HBCK2, you need to point at a built   HBCK2   jar using the   -j   option as in: $ ${HBASE_HOME}/bin/hbase --config /etc/hbase-conf hbck -j ~/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar wherein the above,   /etc/hbase-conf   is where the hbase deployment's configuration files resides. The   HBCK2   jar is at   ~/hbase-operator-tools/hbase-hbck2/target/hbase-hbck2-1.0.0-SNAPSHOT.jar. The above command with no options or arguments passed will dump out the   HBCK2   help: usage: HBCK2 [OPTIONS] COMMAND <ARGS> Options: -d,--debug run with debug output -h,--help output this help message -p,--hbase.zookeeper.property.clientPort <arg> port of hbase ensemble -q,--hbase.zookeeper.quorum <arg> hbase ensemble -s,--skip skip hbase version check (PleaseHoldException) -v,--version this hbck2 version -z,--zookeeper.znode.parent <arg> parent znode of hbase ensemble Command: addFsRegionsMissingInMeta <NAMESPACE|NAMESPACE:TABLENAME>... Options:   ... View more

Hello @marianaduran, Not sure what you are looking for exactly. If you want to use HBase REST interface, please refer Cloudera blog page which talks about using the Apache HBase REST Interface here - https://blog.cloudera.com/how-to-use-the-apache-hbase-rest-interface-part-1/ - https://clouderatemp.wpengine.com/blog/2013/04/how-to-use-the-apache-hbase-rest-interface-part-2/    And Cloudera doc page on Configuring and Using the HBase REST API - https://docs.cloudera.com/documentation/enterprise/5-13-x/topics/admin_hbase_rest_api.html  -  ... View more

Hello @hammer75, currently no document suggests the use of BYOK as a backing Keystore.    Cloudera offers the following two options for enterprise-grade key management: Cloudera Navigator Key Trustee Server is a key store for managing encryption keys. To integrate with the Navigator Key Trustee Server, Cloudera provides a custom KMS service, Key Trustee KMS. Hardware security modules (HSM) are third-party appliances that provide the highest level of security for keys. To integrate with a list of supported HSMs, Cloudera provides a custom KMS service, Navigator HSM KMS (see Installing Navigator HSM KMS Backed by Thales HSM and Installing Navigator HSM KMS Backed by Luna HSM). Ref: https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hdfs_encryption.html#concept_hsm_kms_solution    So  HDFS Data At Rest Encryption wizard in Cloudera Manager offers below 4 roots of trust for encryption keys: Cloudera Navigator Key Trustee Server Navigator HSM KMS backed by Thales HSM Navigator HSM KMS backed by Luna HSM A file-based password-protected Java KeyStore (not for Prod env) ... View more

If CDH upgrade is not possible immediately, setting the "-log_min_segments_to_retain=2" will only prevent the further corruption.     To do this, go to the Kudu configuration page in Cloudera Manager and add " -log_min_segments_to_retain=2" to the   Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile   option. Note: When CDH upgraded to a version containing a fix, remove   -log_min_segments_to_retain   setting.   However, if the tablet failed to start after setting    "-log_min_segments_to_retain=2" , then we have to find the corrupt tablet on the affected TS and delete it in order to bring the TServer up. ... View more

Hi, It seems you hit KUDU-2233 (https://issues.apache.org/jira/browse/KUDU-2233) which is a nasty bug in Kudu 1.6 reported here - https://docs.cloudera.com/documentation/enterprise/release-notes/topics/kudu_known_issues.html#tsb-298 (please refer the complete stack trace in kudu tserver stderr log) KUDU-2233 is fixed in kudu 1.7.0 which is available from CDH 5.15.0. You may consider upgrading the CDH version to CDH 5.15.0 or later. Refer the release note here - https://docs.cloudera.com/documentation/enterprise/release-notes/topics/kudu_release_notes.html#relnotes_5_15_0 Regards, PDas ... View more