Cloudera Community
gael__urbauer
user rank
Explorer
My Accepted Solutions
Show All

Re: Limit Hive query ressource consumption in Yarn

by Explorer in Support Questions
‎09-07-2023 12:32 AM
‎09-07-2023 12:32 AM
Hi tj, I knew already f this options but sadly as mentionned in my post all the queries are ran by the same service user. Hence I have no way to use the user limit factor. I would like to have  a query limit factor in hive or some way to prevent one query to use too much capacity even if available. Gael ... View more

Re: High CPU Load of hadoop process

by Explorer in Support Questions
‎08-24-2023 01:08 AM
‎08-24-2023 01:08 AM
BTW strange to locate datanode in kubernetes as pods are usually used for stateless tasks and a datanode is almost exclusively statefull by nature as it keeps data from HDFS ... View more

Re: High CPU Load of hadoop process

by Explorer in Support Questions
‎08-24-2023 01:06 AM
‎08-24-2023 01:06 AM
The location depends from the distribution and can be changed in the configuration. See this article that gives guidelines to find out for YARN nodemanager. Solved: where are hadoop log files ? - Cloudera Community - 115681 And don't forget to  replace YARN by HDFS if needed as you are looking for the data node service of HDFS and not yarn nodemanager. ... View more

Re: High CPU Load of hadoop process

by Explorer in Support Questions
‎08-23-2023 04:56 AM
‎08-23-2023 04:56 AM
Hi Noel,   The process you are pointing is the nodemanager see the last parameter of java command that is the starting class org.apache.hadoop.hdfs.server.datanode.DataNode  I never experienced such problem with this component. Perhaps you should review the logs from this component and it is retrying something that fails in an infinite loop. open /var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-<host-name>.log.out on this server node to check it. ... View more

Limit Hive query ressource consumption in Yarn

by Explorer in Support Questions
‎08-22-2023 02:39 AM
‎08-22-2023 02:39 AM
Hi,   I face the following problem. I have a service user say "service_dwh" used for our datawarehouse that queries heavily our data reservoir using hive queries. I have had some cases where due to the query and/or to missing statistics a single hive query could take 100% of the resources available for the "service_dwh" user. I couldn't find a way using capacity scheduler, queues and user limit factor to prevent a single application from taking all the resources for a very long time. Traditional DBMS have mechanism that throttle job resources based of the job duration. That way a long (big) job can't monopolyze ressources for new and potentially shorter jobs for too long. ... View more
Labels:

Re: Troubleshooting CM Agent won't connect to CM S...

by Explorer in Support Questions
‎02-10-2022 04:59 AM
‎02-10-2022 04:59 AM
I finally solved the problem with the help of the support. They redirected me to Manually Configuring TLS Encryption for Cloudera Manager | 6.3.x | Cloudera Documentation In the section 6 I had a certificate with only the "TLS Web Server Authentication"  usage. I missed the "TLS Web Client Authentication" Usage.   ... View more

Troubleshooting CM Agent won't connect to CM Serve...

by Explorer in Support Questions
‎02-09-2022 10:39 PM
‎02-09-2022 10:39 PM
Hi, I am provisionning a CDP 7.1.5 cluster with CM 7.2.4  and encountered a connexion problem between the agents and cloudera manager server after tls setup through API. There is nothing in the server log and there is the following error in the agent log. Could you help me understand where the problem stands ? Activating further logging or doing some tests upon the provided certificates ? Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1430, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 185, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: sslv3 alert certificate unknown Regards. ... View more
Labels:

Re: timeline server memory leak?

by Explorer in Support Questions
‎05-27-2021 02:38 AM
‎05-27-2021 02:38 AM
Same problem for us today. 19G yesterday and 11G today after restart. Is there a patch available ? ... View more

Re: Ambari's hive config references hive.metastore...

by Explorer in Support Questions
‎01-18-2017 05:10 PM
‎01-18-2017 05:10 PM
had the same problem on IBM BigInsight 4.1 You saved my day ... View more
Contact Me
Online
Offline
Last Visited
‎01-18-2024 04:25 AM
Community Statistics
Member Since ‎01-18-2017 05:07 PM
Last Visited ‎01-18-2024 04:25 AM
Posts 9