Member since
02-15-2020
18
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3007 | 01-25-2021 09:36 AM |
05-02-2021
09:36 PM
Thanks @Daming Xue . Surely ill look into upgrading the CM version. But for the time being, do we have any other solution as this is in production. We don't want to take risks by directly upgrading the production manager service. There are so many entities that are being monitored ex:- impala, yarn, etc, but we use only Kafka, and mirror maker. Is there any ways to disable it or does the service are necessary for running kafka clusters?.
... View more
05-02-2021
01:24 AM
Thank you @Daming Xue . This is perfect lead. Upon checking, i can see we are monitoring around 3819926 entities in the cloudera manager from the three different kafka cluster. Can you help in avoiding creating so many entities. So that ican help us to free from the alrerts keeps triggering and crashing the service monitor roles. Right now, the heap memory size is 10Gb and the non-heap memory size is 12 Gb. But still we do get the heap memory issue. What would be an ideal solution for us to fix the issues here. We have increased the Heap memory size form 3 GB to 10 Gb till now. Even we added this parameter in the service monitor and restarted the service, but this didn't help. -XX:+UseG1GC -XX:-UseConcMarkSweepGC -XX:-UseParNewGC And the error from the Service monitor log is 2021-05-02 08:24:22,106 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 33651ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=195ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=33878ms
... View more
04-26-2021
09:18 AM
Hi, I've been using cloudera manager for the last one year with running of 20+ nodes. Recently i started to see heap memory size issue in Service monitor roles. I've increased from 3 to 4, then 4 to 5, and then 5 to 6 GB.But still i sometime get the service monitor crashed and restarted. During the time, the entire dashboard seems bad. What i need to do here to fix the issue?. Logs are 2021-04-26 16:10:34,938 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 20583ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=182ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20877ms 2021-04-26 16:11:34,862 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 19870ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=131ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20228ms 2021-04-26 16:12:35,132 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 20427ms: GC pool 'G1 Young Generation' had collection(s): count=3 time=149ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20733ms 2021-04-26 16:13:36,415 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 19008ms: GC pool 'G1 Young Generation' had collection(s): count=1 time=104ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=19381ms Could you please help me on this?.
... View more
Labels:
- Labels:
-
Cloudera Manager
02-16-2021
09:01 PM
Thank you @GangWar . This helps. My only worry is the backend services that are being manged by Cloudera client agent should not be affected. How I can do the command line task for the services? Can you please do the needful by giving some examples or a link to go-through?
... View more
02-11-2021
09:25 PM
Hi, I have a strange doubt in using Cloudera manager express edition. In any scenario, the manager might go down, or the DB can get crashed or the some external issue that put the cloudera down. In such case, the backend services will not affect, but how to manage those services/roles?. Have anyone had this issue. I wonder if one-day manager node goes down, what will we do for the other services?.
... View more
Labels:
- Labels:
-
Cloudera Manager
01-25-2021
09:36 AM
Thanks for the reply, @GangWar. The issue is sorted out now. As you said, the issue got fixed after checking the config.ini. The inter communications between the client and server were made to do over TLS, but we disabled the TLS/SSL in UI and for all agents. This was the main cause, as the heartbeat tried to get a response, but it didnt. Changing the configuration in Cloudera Manager -> Administration -> “Use TLS Encryption for Agents” should be unchecked, and restarting the Cloudera Manager fixed the issue.
... View more
01-21-2021
03:23 AM
Hi, I'm not able to start the management service after the reboot of the manager node. I'm not sure what happened, everything was working fine until the reboot. [21/Jan/2021 11:03:25 +0000] 15063 MainThread agent ERROR Heartbeating to ip-172-31-203-46.ec2.internal:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1399, in _send_heartbeat response = self.requestor.request('heartbeat', heartbeat_data) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request return self.issue_request(call_request, message_name, request_datum) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request call_response = self.transceiver.transceive(call_request) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive result = self.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message framed_message = response_reader.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message raise ConnectionClosedException("Reader read 0 bytes.") ConnectionClosedException: Reader read 0 bytes. The error message in UI is "Role not started due to unhealthy host"
... View more
Labels:
- Labels:
-
Cloudera Manager
04-18-2020
07:43 AM
Thank you @TonyStank . This helps me.
... View more
03-18-2020
12:34 AM
Hi,
I've installed Cloudera Manager with Kafka and Zookeeper. I'm trying to build monitoring along with Prometheus and Graffana. Do we have any inbuilt feature to do in an easy way? or any leads, please.
... View more
Labels:
03-01-2020
09:44 PM
Thank You. This helped a lot. I wrote the scripts for the basic functionalities.
... View more