Member since
03-22-2017
52
Posts
12
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
843 | 10-21-2021 12:49 AM | |
622 | 04-01-2021 05:31 AM | |
666 | 03-30-2021 04:23 AM | |
1396 | 03-23-2021 04:30 AM | |
1820 | 03-05-2021 04:33 AM |
10-22-2021
12:25 AM
@PabitraDas The objective is to copy data between two distinct clusters
... View more
10-21-2021
12:49 AM
@DA-Ka You need to use HDFS Find tool "org.apache.solr.hadoop.HdfsFindTool" for that purpose. Refer below links which suggests some method to fid the old Files. - http://35.204.180.114/static/help/topics/search_hdfsfindtool.html However, t he search-based HDFS find tool has been removed and is superseded in CDH 6 by the native "hdfs dfs -find" command, documented here: https://hadoop.apache.org/docs/r3.1.2/hadoop-project-dist/hadoop-common/FileSystemShell.html#find
... View more
08-31-2021
11:48 PM
Will formatting zkfc and restarting namenode work as this issue is basically due to communication failure between HealthcheckRPC of zkfc and local namenode.
... View more
03-30-2021
08:43 AM
@abagal / @PabitraDas Appreciate all your assistance / inputs on this. Thanks Wert
... View more
03-30-2021
04:23 AM
1 Kudo
Hello @Amn_468 Please note that, you get the block count alert after hitting the warning/critical threshold value set in HDFS Configuration. It is a Monitoring alert and doesn't impact any HDFS operations as such. You may increase the monitoring threshold value in CM ( CM > HDFS > Configurations > DataNode Block Count Thresholds) However, CM monitors the block counts on the DataNodes is to ensure you are not writing many small files into HDFS. Increase in block counts on DNs is an early warning of small files accumulation in HDFS. The simplest way to check if you are hitting small files issue is to check the average block size of HDFS files. Fsck should show the average block size. If it's too low a value (eg ~ 1MB), you might be hitting the problems of small files which would be worth looking at, otherwise, there is no need to review the number of blocks. [..] $ hdfs fsck / .. ... Total blocks (validated): 2899 (avg. block size 11475601 B) <<<<< [..] Similarly, you can get the average file size in HDFS by running a script as follows: $hdfs dfs -ls -R / | grep -v "^d" |awk '{OFMT="%f"; sum+=$5} END {print "AVG File Size =",sum/NR/1024/1024 " MB"}' The file size reported by Reports Manager under "HDFS Reports" in Cloudera Manager can be different as the report is extracted from >1hour old FSImage (not a latest one). Hope this helps. Any question further, feel free to update the thread. Else mark solved. Regards, Pabitra Das
... View more
03-23-2021
08:03 PM
1 Kudo
Thank you everyone. My problem is just uuid file in line feed.
... View more
03-22-2021
04:50 AM
Hello @pauljoshiva You need to add the new nodes with a new config group. One set of DNs in default config group (where the storage directories are laid from /hdp/hdfs01 - /hdp/hdfs09) and anotehr set of DNs in new config group (with directories /hdp/hdfs01, /hdp/hdfs02, /hdp/hdfs03). That way you can have all DNs added to cluster with 2 separate config groups.
... View more
03-16-2021
03:08 AM
Hello @Monds you can recover the lease on the file, running below command: #hdfs debug recoverLease -path <path-of-the-file> [-retries <retry-times>] This command will ask the NameNode to try to recover the lease for the file (successfully close the file if there are still healthy replicas) Ref: https://blog.cloudera.com/understanding-hdfs-recovery-processes-part-1/
... View more
03-15-2021
04:40 AM
Hello @Babar Thank you for resolving the issue and marking the thread as solved. Glad to know that you identify the problem and resolved it. Please note HDFS-14383 (Compute datanode load based on StoragePolicy) has been included in the recent release of CDP 7.1.5 and 7.2.x
... View more
03-14-2021
04:42 PM
Thank you for helping me!!! 🙂
... View more
03-05-2021
04:40 AM
Hello @PabitraDas , that's the information I needed thanks a lot! br
... View more
03-04-2021
10:38 AM
Hi @dv_conan, Similar issue is addressed here - https://community.cloudera.com/t5/Support-Questions/failed-to-execute-command-install-yarn-mapreduce-framework/td-p/301804 Please refer and make the necessary changes to directory permissions and let us know if that helped you.
... View more
03-04-2021
10:30 AM
Hello @samglo , Please note Solr CDCR is not supported in CDP yet. Refer to Cloudera blog on Solr CDCR (Cross Data Center Replication) support: - https://blog.cloudera.com/backup-and-disaster-recovery-for-cloudera-search/ Solr CDCR The future holds the promise of a Solr to Solr replication feature as well, a.k.a. CDCR . This is still maturing upstream and will need some time to further progress before it can be considered for mission critical production environments. Once it matures we will evaluate its value in addition to all our existing options of recovery for Search. The above solutions, presented in this blog, are production-proven and provides a very good coverage along with flexibility for today’s workloads. However, you can refer apache document on Solr CDCR below for some information about setup: - https://solr.apache.org/guide/6_6/cross-data-center-replication-cdcr.html or Cloudera Community article - https://community.cloudera.com/t5/Community-Articles/How-to-setup-cross-data-center-replication-in-SolrCloud-6/ta-p/247945
... View more
03-04-2021
10:06 AM
Hello @nj20200 It seems there is an older/previous version of openssl-devel package ( openssl-libs-1.0.2k-19.el7.x86_64) is installed, which is causing the installation failure of new version openssl-devel package ( openssl-devel-1.0.1e-60.el7.x86_64). So instead of installing the package, update the openssl-devel package by running "#yum update openssl-devel with -force option" or just remove the previous package and install the new version of openssl-devel package.
... View more
03-02-2021
09:23 AM
Yes @PabitraDas , it's accessible now. It would be helpful to have a secondary repository or maintenance windows announced.
... View more
03-02-2021
09:09 AM
Hello @kolli_sandeep , it seems the failover controllers are down in the cluster. Please follow the steps here [1] and start the Failover Controller roles which will transition the NameNdoes to Active/Standby state. You need to follow below steps; Stop the FailoverController Roles under HDFS > Instances page Remove the HA state from ZK. On a ZooKeeper server host, run zookeeper-client. Execute the following to remove the configured nameservice. This example assumes the name of the nameservice is nameservice1. You can identify the nameservice from the Federation and High Availability section on the HDFS Instances tab: rmr /hadoop-ha/nameservice1 (If you don't see any znode /hadoop-ha in ZK znode list, skip the step) After removing the HA znode in ZK, Go to CM and Click the HDFS > Instances > Federation and High Availability > Actions Under Actions menu, Select Actions > Initialize High Availability State in ZooKeeper . Then start the Failover Controllers role ( CM > Instances > Select FailoverControllers > Actions for selected > Start) Verify the NameNdoe State and if you don't see the active/standby state of NN, If any failure, just Restart the HDFS service [1] https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_hdfs_ha_enabling.html
... View more
03-01-2021
08:28 PM
Thanks @PabitraDas Alright, done as your suggestion, the cluster is looking good now without the alert Thanks for the link too.
... View more
02-01-2021
10:35 AM
Hello @vvk Please note, while adding/removing the journal nodes from the running cluster, you need to ensure a quorum of journal nodes available for NameNodes. (As cited in the shared document--> NameNode high availability requires that you must maintain at least three, active JournalNodes in your cluster.) It means NameNode ensures at least a quorum of Journal Nodes (2 of 3 journal nodes) available for edits log write at any given point Failing to write edits into a quorum of journal nodes, NameNode is expected to crash (shutdown itself). I believe this could be the scenario in your case. So you need to add new journal nodes first to the cluster before removing the old Journal nodes one by one ensuring a quorum of journal nodes available in the cluster. If you see NN crashed even after edits log write was successful on a quorum of JNs, then we need to check the NN log for any other issues. Thank you
... View more
12-08-2020
06:26 AM
Hello @tuk If the Post by Pabitra assisted you, Kindly mark the Post as Solution. If you utilised any other approach, Kindly share the details in the post as well. Thanks, Smarak
... View more
11-11-2020
02:06 AM
Hello @Amn_468 Since you reported the DN Pause time, I spoke/referred about DN heap only. The block counts on most of the DN seems >6Millions, hence would suggest to increase the DN heap to 8GB (from current value of 6GB) and perorm a rolling restart to bring the new heap size into effect. There is no straight forward way to say you hit the small file problem but if your average block size is few MB or less than a MB in size, it is an indication that you are storing/accumulating small files in HDFS. Simplest way to determine small files in cluster is to run fsck. Fsck should show the average block size. If it's too low a value (eg ~ 1MB ), you might be hitting the problems of small files which would be worth looking at, otherwise, there is no need to review the number of blocks. [..] $ hdfs fsck / .. ... Total blocks (validated): 2899 (avg. block size 11475601 B) <<<<< [..] You may refer belwo links for your help on dealing with small files. - https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/ - https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-located-in-a-large/ta-p/247253
... View more
11-09-2020
10:20 PM
Thanks, I'm able to access the Hadoop CLI after commenting out the line.
... View more
11-09-2020
09:42 AM
Hello @Masood, I believe you are asking the commands to run in order to determine the active NN apart from CM UI ( CM > HDFS > Instance > NameNode) From CLI you have to run couple of commands to detemrine the Active/Standby NN List the namenode hostnames # hdfs getconf -namenodes c2301-node2.coelab.cloudera.com c2301-node3.coelab.cloudera.com Get nameservice name # hdfs getconf -confKey dfs.nameservices nameservice1 Get active and standby namenodes # hdfs getconf -confKey dfs.ha.namenodes.nameservice1 namenode11,namenode20 # su - hdfs $ hdfs haadmin -getServiceState namenode11 active $ hdfs haadmin -getServiceState namenode20 standby Get active and standby namenode hostnames $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode11 c2301-node2.coelab.cloudera.com:8020 $ hdfs getconf -confKey dfs.namenode.rpc-address.nameservice1.namenode20 c2301-node3.coelab.cloudera.com:8020 If you want to get the active namenode hostname from hdfs-site.xml file, you can go through following python script in github – https://github.com/grakala/getActiveNN . Thank you
... View more
11-09-2020
09:06 AM
Hello @AlexP Ref: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep Referring to HDFS document, answers to your questions are inline. [Q1.] How to estimate how much time would this command take for a single directory (without -w)? [A1.] It depends upon the numbr of files in the directory. If you are running setrep against a path which is a directory, then the command recursively changes the replication factor of all files under the directory tree rooted at path. The time varies dependsing on the file count under the path/directory. [Q2.] Will it trigger a replication job even if I don't use the '-w' flag? [A2.] Yes, replication will trigger without -w flag. However, it is good practice to use -w to ensure all files are having required replication factor set prior to command exits. Please note, the -w flag requests that the command wait for the replication to complete. Though use of -w potentially takes a long time to complete the command but it gurantees the replication factor changed to the specified value. [Q3.] If yes, does it mean that the NameNode will actually start deleting 'over-replicated' blocks of all existing files under a particular directory? [A3.] Yes, your understanding is correct. The additonal 1 replica of the block will mark the block as over-replicated and same will be deleted from cluster. This action will be performed for each files under the directory path keeping only 2 replicas of the file blocks. Hope this helps.
... View more
10-10-2020
11:35 AM
1 Kudo
@mike_bronson7 Always stick to the Cloudera documentation. Yes !!! there is no risk in running that command I can understand your reservation.
... View more
09-30-2020
10:14 PM
Hello @vincentD Please review the stdout and stderr of the DN which going down frequently. You can navigate to CM > HDFS > Instance > Select the DN which went down > Processes > click on stdout/stderr atthe bottom of the page. I am asking to verify stdout/stderr suspecting an OOM error (due to java heap running out of memory) leading to the DN exit/shutdown abruptly. If the DN exit is due to OOM Error, please increase the DN heap size to adequate value to get rid off teh issue further. DN heap sizing rule of thumb says: 1 GB heap memory for 1Million blocks. You can verify your block counts on each DN by navigating to CM > HDFS > NN Web UI > Active NN > DataNode and you can see the DN stats on that page showing block counts and disk usage etc..
... View more
09-30-2020
09:02 AM
Thank you for verifying!
... View more
09-29-2020
02:01 PM
Hello @marianaduran, Not sure what you are looking for exactly. If you want to use HBase REST interface, please refer Cloudera blog page which talks about using the Apache HBase REST Interface here - https://blog.cloudera.com/how-to-use-the-apache-hbase-rest-interface-part-1/ - https://clouderatemp.wpengine.com/blog/2013/04/how-to-use-the-apache-hbase-rest-interface-part-2/ And Cloudera doc page on Configuring and Using the HBase REST API - https://docs.cloudera.com/documentation/enterprise/5-13-x/topics/admin_hbase_rest_api.html -
... View more
09-29-2020
01:28 PM
Hello @hammer75, currently no document suggests the use of BYOK as a backing Keystore. Cloudera offers the following two options for enterprise-grade key management: Cloudera Navigator Key Trustee Server is a key store for managing encryption keys. To integrate with the Navigator Key Trustee Server, Cloudera provides a custom KMS service, Key Trustee KMS. Hardware security modules (HSM) are third-party appliances that provide the highest level of security for keys. To integrate with a list of supported HSMs, Cloudera provides a custom KMS service, Navigator HSM KMS (see Installing Navigator HSM KMS Backed by Thales HSM and Installing Navigator HSM KMS Backed by Luna HSM). Ref: https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hdfs_encryption.html#concept_hsm_kms_solution So HDFS Data At Rest Encryption wizard in Cloudera Manager offers below 4 roots of trust for encryption keys: Cloudera Navigator Key Trustee Server Navigator HSM KMS backed by Thales HSM Navigator HSM KMS backed by Luna HSM A file-based password-protected Java KeyStore (not for Prod env)
... View more
09-29-2020
01:12 PM
I used the systemd. It worked great. Again, I appreciate everyone's help!
... View more