Skip to main content
sharathkumar13
sharathkumar13
Gold ⭐️
April 26, 2021
Solved

Service monitor keeps crashing

Hi, 

 

I've been using cloudera manager for the last one year with running of 20+ nodes. Recently i started to see heap memory size issue in Service monitor roles. I've increased from 3 to 4, then 4 to 5, and then 5 to 6 GB.But still i sometime get the service monitor crashed and restarted. During the time, the entire dashboard seems bad. What i need to do here to fix the issue?. 

 

Logs are

2021-04-26 16:10:34,938 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 20583ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=182ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20877ms
2021-04-26 16:11:34,862 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 19870ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=131ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20228ms
2021-04-26 16:12:35,132 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 20427ms: GC pool 'G1 Young Generation' had collection(s): count=3 time=149ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=20733ms
2021-04-26 16:13:36,415 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 19008ms: GC pool 'G1 Young Generation' had collection(s): count=1 time=104ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=19381ms

 

Could you please help me on this?. 

Daming Xue
Daming XueAnswer
Gold ⭐️
April 27, 2021

Hello

 

Depends on the cluster size and monitored entities, certain resources are recommended

 

Please refer to the below link and check whether your resource allocation is in-line with the recommendations

 

https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-service-monitor-requirements.html

View in thread

5 replies

Daming Xue
Gold ⭐️
Daming XueAnswer
Gold ⭐️
April 27, 2021

Hello

 

Depends on the cluster size and monitored entities, certain resources are recommended

 

Please refer to the below link and check whether your resource allocation is in-line with the recommendations

 

https://docs.cloudera.com/cdp-private-cloud/latest/release-guide/topics/cdpdc-service-monitor-requirements.html

Translate
sharathkumar13
sharathkumar13Author
Gold ⭐️
May 2, 2021

Thank you @Daming Xue . This is perfect lead. 

 

Upon checking, i can see we are monitoring around 3819926 entities in the cloudera manager from the three different kafka cluster. Can you help in avoiding creating so many entities. So that ican help us to free from the alrerts keeps triggering and crashing the service monitor roles. 

 

Right now, the heap memory size is 10Gb and the non-heap memory size is 12 Gb. But still we do get the heap memory issue. What would be an ideal solution for us to fix the issues here. We have increased the Heap memory size form 3 GB to 10 Gb till now. 

 

Even we added this parameter in the service monitor and restarted the service, but this didn't help. 

-XX:+UseG1GC -XX:-UseConcMarkSweepGC -XX:-UseParNewGC

And the error from the Service monitor log is

2021-05-02 08:24:22,106 WARN com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 33651ms: GC pool 'G1 Young Generation' had collection(s): count=2 time=195ms, GC pool 'G1 Old Generation' had collection(s): count=1 time=33878ms

Translate
Daming Xue
Gold ⭐️
Daming Xue
Gold ⭐️
May 3, 2021

Hello

 

Please look out for Cloudera Manager 7.4.1 update, which comes with the fix to the issue you are facing

 

The issue is because producer client ids that are not configured correctly and are generating huge metrics from anonymous producers

 

The solution proposed in CM 7.4.1 is to make the Kafka producer metric whitelist configurable in the CM UI

Translate
Terms & ConditionsAccessibility statement