About snm1523

snm1523 · ‎03-20-2020

Hello, If we Go to Clusters > Kudu > Click Actions, Run Kudu Rebalancer Tool, will it distribute data for overall Kudu or just Kudu Master or Kudu T-Servers. Target is to distribute currently uneven data, evenly acrossall nodes. If above does not work, is there any other way to distribute / balance data evenly across all kudu nodes. Also, do we need any downtime while rebalancing is in progress unlike HDFS. Overview of our infrastructure: 3 Kudu Masters 9 Tablet Servers Kudu 1.7.0-cdh5.16.2/ CM 5.16.2 Request some advice / assistance on the same. Thanks Snm1523

snm1523 · ‎02-14-2020

Hello, We are in the process of deploying CDP on Azure for our test system and later would move to Production. However, we are stuck with the port requirements for this deployment. As per the guidance from Cloudera, initially we were given a huge list of 7k+ port numbers that are required to be opened. That certainly would not pass network restrictions and hence, post further deep dive and discussions, we are currently at a point of getting only 6015 port number opened. We have also requested for a better explanation of why these 7k+ port numbers or just 6015 is required. However, while we are waiting for a response, I thought of to seek suggestions from the community, if anybody had deployed CDP on Azure and might have come across this. Would be great if anyone could advise on below: 1. What is the real need of 6015? I did not find any reference of this port in any of the community documentations by Cloudera 2. Will we really need a list of 7k+ ports to be opened? or anybody has quick list of required ports to be opened and can share that. 3. What are the other best practices / precautions you would have considered if deployed CDP on Azure or any other Cloud provider. Thanks snm1523

snm1523 · ‎01-24-2020

Perfect explanation. Thank you @EricL. This certainly helped.

snm1523 · ‎01-23-2020

Hello, While browsing through various articles related to Cloudera Manager, I came across an article which talks about exporting Cloudera Manager configs using a script. In this a wget from " ${SCM_URL} /cmf/exportCLI" url is doing the trick. Being fairly new with playing APIs, I tried to read across somethings related to CM APIs, however, did not find much. Would be great if someone could help me to understand which exact configurations are exported with the mentioned API. Apologies if seems to be a basic question. However, thought of asking community instead of creating confusions for myself. Thanks snm1523

snm1523 · ‎01-23-2020

Hello @GangWar, Thank you for the reply, Contacting Cloudera is certainly a best option and would most likely go for it. However, currently the project is in discussion and I was just trying to get views / thoughts from the community on such a scenario if anybody had ever come across. Thanks snm1523

snm1523 · ‎01-14-2020

Hello @Lewis , Thank you for the reply. I have a scenario, we are on multiple cdh 5.x versions on multiple clusters. Along with a small piece on Altus. Goal is to get everything on Azure, in such case would your opinion be to setup a fresh CDP on Azure and then move data and applications cluster by cluster to CDP? Thanks snm1523

snm1523 · ‎01-13-2020

Hello, Do we have a straight migration / upgrade path from CDH 5.x or 6.x to CDP on Azure? I tried to get this info in various forums / Cloudera documentations, however, was unable to find anything. Probably there is no such path or I may have missed something. Still, looking for a reconfirmation or guidance or advise on this. Please help. Thanks Snm1523

snm1523 · ‎09-27-2019

Hello All, Need suggestions / best practices / guidance to deploy below HBase replication scenario: We have 2 running HBase clusters which are in sync with each other (Bi directional HBase replication configured and enabled). With this it ensures that both HBase clusters have up to date data in HBase tables. We have a backup cluster deployed and the idea is to setup replication from both HBase clusters to Backup cluster. So ultimately, we will have 2 Active Masters of which data will be replicated to 1 Slave. If we configure HBase replication jobs for both master and keep them running, it could be the case of duplication of data (as per my understanding and logic). Would be great if anyone could share their experiences or guide further on to achieve this. Thanks Snm1523

snm1523 · ‎06-03-2019

Hello, Is there any way we can configure Cloudera Manager to send an email if utilization of CPU or Memory of a particular daemon / node (for eg. Impala daemon) crosses a specific limit (for eg. 80%)? Please advise. Thanks snm1523

snm1523 · ‎06-03-2019

Hello Harsh, Thank you for the help on this. I was able to identify some information that helped here. Will come back in case need further help. Will accept your reply as Solution. 🙂 Thanks snm1523

snm1523 · ‎03-07-2019

Thank you for the reply Harsh J. Would you be able to please help me with any quick command / script to identify avoidable open files or files stuck in some process using 'lsof' and guide further actions to take? I tried running a generic 'lsof | grep java' but it obviously gave me a huge list of files and became a bit difficult to get relevant information. Thanks snm1523

snm1523 · ‎03-06-2019

Hello All, I am looking for some best practices or recommendations to set a best possible value for rlimit_fds (Maximum Process File Descriptors) property. Currently, it is set to default i.e. 32768 and we are getting File Descriptor Threshold alerts. We would first like to look for a best possible value for rlimit_fds. Is there a formulae or a practice or few checks that can be performed to determine a best value? Thanks snm1523

snm1523 · ‎02-25-2019

Hello gzigldrum, Thank you for the guidance. This certainly helps. I will go through both the KB articles you shared and also review the Navigator Server and CM Agent logs for audit health warnings. Will comeback with the findings. In the meantime, any luck with HIVEMETASTORE_CANARY_HEALTH and NODE_MANAGER_WEB_METRIC_COLLECTION? Thanks snm1523

snm1523 · ‎02-22-2019

Hello Gzigldrum, Thank you for the reply. Below are the exact health messages from CM for each alert: HIVEMETASTORE_CANARY_HEALTH: The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a partition in the table it created. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to drop the table it created. REGION_SERVER_AUDIT_HEALTH: The health test result for REGION_SERVER_AUDIT_HEALTH has become bad: There is a problem processing audits for REGIONSERVER. IMPALAD_QUERY_MONITORING_STATUS: he health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 1 error(s) seen monitoring executing queries, and 0 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any. HIVESERVER2_SCM_HEALTH: The health test result for HIVESERVER2_SCM_HEALTH has become bad: This role s process is starting. This role is supposed to be started. NAME_NODE_AUDIT_HEALTH: The health test result for NAME_NODE_AUDIT_HEALTH has become bad: There is a problem processing audits for NAMENODE. NODE_MANAGER_WEB_METRIC_COLLECTION: The health test result for NODE_MANAGER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent is not able to communicate with this role s web server. These health alerts occur in every 2-3 days in a time frame of 2-3 hours creating around 5-10 tickets during each interval. We have also checked from network side to verify if there was a network outage or a glitch in those windows, however, no luck. We have tried to diagnose through logs of each alert, but haven't found anything interesting. Hence, looking for some more guidance on what additional could be checked to identify a root cause of these. Thanks snm1523

snm1523 · ‎02-21-2019

Thank you for the quick replies, Gzigldrum. I have created a separate post explaining the alerts we get. Below is the link to the same. Flooded with failed health test alerts For every alert we get an incident in BMC as HP OVO is configured to generate an incident and assign it to us. Every morning we start with close to around 15-20 such incidents followed by 15-20 in rest of the day. Would be great if you could please post some suggestions to troubleshoot those alerts and help in permanently fixing them. Note: I will accept your previous reply as solution to this post. Thanks snm1523

snm1523 · ‎02-21-2019

Hello Gzigldrum, Thank you for the reply. However, the instructions provided in the KB article and values suggested by you are already applied. Please advise. Thanks Snm1523

snm1523 · ‎02-21-2019

Hello All, Need some suggestions on the exact reason of below heath tests getting failed multiple times, almost 3 times every day and generates atleast 3-4 alerts each time. 1. HIVEMETASTORE_CANARY_HEALTH 2. REGION_SERVER_AUDIT_HEALTH 3. IMPALAD_QUERY_MONITORING_STATUS 4. HIVESERVER2_SCM_HEALTH 5. NAME_NODE_AUDIT_HEALTH 6. NODE_MANAGER_WEB_METRIC_COLLECTION Any help or suggestion to permanently fix these alerts would of great help. Also, if anyone could also guide to reach root cause of this would also be helpful Thanks snm1523

snm1523 · ‎02-21-2019

Hello All, We have a HP OVO monitoring tool monitoring all the alerts from Cloudera Manager and raising a BMC Remedy Incident accordingly. However, we are at times flooded with these monitoring alert tickets and when it is immediately checked in the cluster, everything looks green. When we dig in for detailed analysis and check logs of the respective alert, we do not have anything major and also the service is green. It looks like it heals by itself. However, this flood of tickets, raises concerns with Management and questions for a root cause, which we do not have in reality and also unable to find anything as it was auto fixed with no special traces behind. I am looking out for recommendations / best practices on how exactly this should be setup, so we get only the actual / required alerts. Is there any configuration we need to do in CM or there is something that can be configured in HP OVO. Any suggestions would be of a great help. Also, if anyone has a suggestion that should be checked while troubleshooting these alerts would also be welcomed. Thanks snm1523

snm1523 · ‎02-18-2019

Thank you gzigldrum. This certainly helps. Regards, snm1523

snm1523 · ‎02-15-2019

Hello, I would like to know the various activities performed by respective users on Cloudera Manager. For example, which user restarts a service or moves a cluster / service to maintenance mode. Would be great if any one could share some information on this. Thanks snm1523

snm1523 · ‎12-04-2018

Hello TK, Were you able to fix this? Would be great if you could please share the solution as I have the same problem. Thanks Snm1523

snm1523 · ‎09-19-2018

Hello @bgooley, Thank you for the guidance. Will attempt the suggested suggestion and come back. Thanks snm1523

snm1523 · ‎09-11-2018

Hello, I am trying to deploy a POC cluster using Path A of Cloudera Manager installation documentation. However, it is failing with below message in the logs: cat 5.install-cloudera-manager-server-db-2.log Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-server-db-2.x86_64 0:5.15.1-1.cm5151.p0.3.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 --> Finished Dependency Resolution Error: Package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 (cloudera-manager) Requires: postgresql-server >= 8.4 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest I also tried to use local repository and point .bin file using --skip_repo_package=1. However, Was unable to find packages for PostgreSQL 8.4 any where, so created repo of 9.6 instead. CM at the first place hunts for PostgreSQL 8.4, and then finds 9.6 in the repo, however, fails at dependency check. It still needs few dependent packages (5-6) to proceed. Before I download each package and put it in repo, seeking suggestions / help from the community on why is the installation with Cloudera repos (installation path A) failing with that message. Thanks snm1523

snm1523 · ‎09-11-2018

Hello, I am trying to deploy a POC cluster using Path A of CM Installation. It is failing with below message from logs: cat 5.install-cloudera-manager-server-db-2.log Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-server-db-2.x86_64 0:5.15.1-1.cm5151.p0.3.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 --> Finished Dependency Resolution Error: Package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 (cloudera-manager) Requires: postgresql-server >= 8.4 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest I also attempted to create a local repository of both CM and PostgreSQL and executed .bin with --skip_repo_package=1. However, I was unable to find PostgreSQL 8.4 package for CentOS / RedHat 7 any where, so downloaded and created repo of PostgreSQL 9.6. CM was further looking for 9.6 dependencies, which was again a list of 5-6 packages. Not sure, how to proceed, please help. Thanks snm1523

snm1523 · ‎07-26-2018

Hi, As per my understanding, CM performs canary tests for each of the enabled and active service in the cluster to ensure if those services are healthy. However, I saw few errors last week related to Hive Metastore Canary test which failed because Hue was unable to create a test database in Hive Metastore. We do not use Hue actively and hence, it has been stopped. After I started Hue service the error was gone and everything reported healthy. Did canary test actually happened on a stopped Hue service or there is something I might have missed? If CM still performs a canary test on stopped services, is there some mis configuration or CM is designed to do this? Please advise. Thanks snm1523

snm1523 · ‎04-05-2018

Hi Jaimie, Just did a little google and found below things to be set. This may resolve the issue: set hive.tez.container.size=2048; set hive.tez.java.opts=-Xmx1700m; --set this 80% of hive.tez.container.size Please give it a try if you feel confident on this. Thanks snm1502

snm1523 · ‎04-05-2018

Hi Brit, One of the log refer to Regex Serde issue: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found You may also verify if the serde is added and available in the mentioned path in Metastore before this job is executed. Thanks snm1502

snm1523 · ‎03-21-2018

Hello, Do we have a specific guide or a standards document any where (or may be anyone might have created based on their experience) that would provide the best practices / approaches / recommendations for various services that we install in a Cloudera Cluster. For eg. 1. Recommendation on Number of executors should be configured while setting up Spark 2. Best practices or things one should definitely keep in mind while setting up a Cloudera Hadoop cluster with bundled Hadoop. 3. Best practices or recommendations for configurations of Oozie, etc. Any leads on this would be helpful. Thanks snm1529

snm1523 · ‎03-21-2018

Hello All, Replying on an old post. Part A: Did anyone made any success in migrating live Cloudera Management Services from one host to another? Part B: For eg. if I have a Kafka job running that is continuosly streaming the data into HDFS, if I stop Cloudera Management Services, will that job get impacted. My assumption is No, however, would like to have expert comments. Thanks snm1529

snm1523 · ‎09-28-2017

Hello @bgoogley, Thank you for the reply. We have already set the swappiness to 1. Still getting these swappiness warnings. Regards, Snm

Online	Offline
Last Visited	‎03-20-2020 12:45 PM

Date Registered	‎10-29-2015 07:36 PM
Last Visited	‎03-20-2020 12:45 PM
Total Messages Posted	47
Total Kudos Received	8

Re: MKDirs failed to create file

Rebalancing in Kudu

CDP on Azure

Re: Cloudera Manager APIs

Cloudera Manager APIs

Re: CDH to CDP Migration

Re: CDH to CDP Migration

CDH to CDP Migration

HBase replication from Multiple HBase clusters

Report or alert based CPU and Memory usage

Re: Calculate File Descriptor in HBase

Re: Calculate File Descriptor in HBase

Calculate File Descriptor in HBase

Re: Flooded with failed health test alerts

Re: Flooded with failed health test alerts

Re: Best Practice to monitor alerts from Cloudera ...

Re: Best Practice to monitor alerts from Cloudera ...

Flooded with failed health test alerts

Best Practice to monitor alerts from Cloudera Mana...

Re: Cloudera Manager user activity audit

Cloudera Manager user activity audit

Re: Could not get LDAP details for groups in patte...

Re: Cloudera Manager Installation failed with Post...

Cloudera Manager Installation failed with PostgreS...

CM Installation Failed with PostgreSQL 8.4 depende...

Canary Tests in Cloudera Manager

Re: Hive does not run query: error 2

Re: HIVE Query Fails but can preview data

Recommendations for Cloudera Hadoop Installation

Re: Migrate cloudera management service on other h...

Re: Swap Memory Usage for HDFS, Yarn and Impala