Hello @raghurok , Could you please check now and see if you still getting the timeout error. I believe the timeout is due to some network glitch or maintenance activity. I hope you will be able to access it now.  ... View more

Hi @pauljoshiva    Though it is expected to have uniform disk configiuration across datanodes in cluster, you can have two different sets of disk confgiration on DNs. You can have one parition of 2TB size on each disk (3 *2TB =6TB on each DN) even though existing has 1 TB size of partition on each disk across all 9 disks (9*1TB=9TB on each DN).    There will be no issue running DNs with such configiration, but you may see 6TB size DNs are filling faster than 9TB size DNs due to the fact that, NN doesn't consider availabel free space on DNs before writing blocks into it. NameNode picks the DN randomly after evaluating network distnace of the DN from client.   Hope this helps. Thank you    ... View more

Hello @vvk  Please note, while adding/removing the journal nodes from the running cluster, you need to ensure a quorum of journal nodes available for NameNodes. (As cited in the shared document-->  NameNode high availability requires that you must maintain at least three, active JournalNodes in your cluster.) It means NameNode ensures at least a quorum of Journal Nodes (2 of 3 journal nodes) available for edits log write at any given point  Failing to write edits into a quorum of journal nodes, NameNode is expected to crash (shutdown itself). I believe this could be the scenario in your case.    So you need to add new journal nodes first to the cluster before removing the old Journal nodes one by one ensuring a quorum of journal nodes available in the cluster.    If you see NN crashed even after edits log write was successful on a quorum of JNs, then we need to check the NN log for any other issues.    Thank you     ... View more

Hello @tuk    If the Post by Pabitra assisted you, Kindly mark the Post as Solution. If you utilised any other approach, Kindly share the details in the post as well.   Thanks, Smarak ... View more

Hello @Masood, I believe you are asking the commands to run in order to determine the active NN apart from CM UI ( CM > HDFS > Instance > NameNode)   From CLI you have to run couple of commands to detemrine the Active/Standby NN   List the namenode hostnames # hdfs getconf -namenodes c2301-node2.coelab.cloudera.com c2301-node3.coelab.cloudera.com   Get nameservice name # hdfs getconf -confKey dfs.nameservices nameservice1   Get active and standby namenodes # hdfs getconf -confKey dfs.ha.namenodes.nameservice1 namenode11,namenode20   # su - hdfs $ hdfs haadmin -getServiceState namenode11 active $ hdfs haadmin -getServiceState namenode20 standby   Get active and standby namenode hostnames   $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode11 c2301-node2.coelab.cloudera.com:8020   $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode20 c2301-node3.coelab.cloudera.com:8020   If you want to get the active namenode hostname from hdfs-site.xml file, you can go through following python script in github –  https://github.com/grakala/getActiveNN .   Thank you  ... View more

Hello @AlexP    Ref: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep   Referring to HDFS document, answers to your questions are inline. [Q1.] How to estimate how much time would this command take for a single directory (without -w)? [A1.] It depends upon the numbr of files in the directory. If you are running setrep against a path which is a directory, then the command recursively changes the replication factor of all files under the directory tree rooted at path. The time varies dependsing on the file count under the path/directory.    [Q2.] Will it trigger a replication job even if I don't use the '-w' flag? [A2.] Yes, replication will trigger without -w flag. However, it is good practice to use -w to ensure all files are having required replication factor set prior to command exits. Please note, the -w flag requests that the command wait for the replication to complete. Though use of -w potentially takes a long time to complete the command but it gurantees the replication factor changed to the specified value.    [Q3.] If yes, does it mean that the NameNode will actually start deleting 'over-replicated' blocks of all existing files under a particular directory? [A3.] Yes, your understanding is correct. The additonal 1 replica of the block will mark the block as over-replicated and same will be deleted from cluster.  This action will be performed for each files under the directory path keeping only 2 replicas of the file blocks.    Hope this helps.      ... View more

Hello @vincentD Please review the stdout and stderr of the DN which going down frequently. You can navigate to CM > HDFS > Instance > Select the DN which went down > Processes > click on stdout/stderr atthe bottom of the page.     I am asking to verify stdout/stderr suspecting an OOM error (due to java heap running out of memory) leading to the DN exit/shutdown abruptly.     If the DN exit is due to OOM Error, please increase the DN heap size to adequate value to get rid off teh issue further. DN heap sizing rule of thumb says: 1 GB heap memory for 1Million blocks. You can verify your block counts on each DN by navigating to CM > HDFS > NN Web UI > Active NN > DataNode and you can see the DN stats on that page showing block counts and disk usage etc.. ... View more

Hello @marianaduran, Not sure what you are looking for exactly. If you want to use HBase REST interface, please refer Cloudera blog page which talks about using the Apache HBase REST Interface here - https://blog.cloudera.com/how-to-use-the-apache-hbase-rest-interface-part-1/ - https://clouderatemp.wpengine.com/blog/2013/04/how-to-use-the-apache-hbase-rest-interface-part-2/    And Cloudera doc page on Configuring and Using the HBase REST API - https://docs.cloudera.com/documentation/enterprise/5-13-x/topics/admin_hbase_rest_api.html  -  ... View more

Hello @hammer75, currently no document suggests the use of BYOK as a backing Keystore.    Cloudera offers the following two options for enterprise-grade key management: Cloudera Navigator Key Trustee Server is a key store for managing encryption keys. To integrate with the Navigator Key Trustee Server, Cloudera provides a custom KMS service, Key Trustee KMS. Hardware security modules (HSM) are third-party appliances that provide the highest level of security for keys. To integrate with a list of supported HSMs, Cloudera provides a custom KMS service, Navigator HSM KMS (see Installing Navigator HSM KMS Backed by Thales HSM and Installing Navigator HSM KMS Backed by Luna HSM). Ref: https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hdfs_encryption.html#concept_hsm_kms_solution    So  HDFS Data At Rest Encryption wizard in Cloudera Manager offers below 4 roots of trust for encryption keys: Cloudera Navigator Key Trustee Server Navigator HSM KMS backed by Thales HSM Navigator HSM KMS backed by Luna HSM A file-based password-protected Java KeyStore (not for Prod env) ... View more

If CDH upgrade is not possible immediately, setting the "-log_min_segments_to_retain=2" will only prevent the further corruption.     To do this, go to the Kudu configuration page in Cloudera Manager and add " -log_min_segments_to_retain=2" to the   Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile   option. Note: When CDH upgraded to a version containing a fix, remove   -log_min_segments_to_retain   setting.   However, if the tablet failed to start after setting    "-log_min_segments_to_retain=2" , then we have to find the corrupt tablet on the affected TS and delete it in order to bring the TServer up. ... View more