SERVICE_MONITOR_AGGREGATION_RUN_DURATION has become bad

[#1]

SERVICE_MONITOR_AGGREGATION_RUN_DURATION


The health test result for SERVICE_MONITOR_AGGREGATION_RUN_DURATION has become bad: The last metrics aggregation run duration is 33.9 second(s). Critical threshold: 30 second(s).

 

Service Monitor  File /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR.log.out


8:35:51.569 PM	WARN	Groups	Potential performance problem: getGroups(user=hue) took 8024 milliseconds.

8:36:21.270 PM	WARN	JvmPauseMonitor	Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 18430ms: no GCs detected.

8:36:21.558 PM	WARN	EnterpriseService com.cloudera.cmf.PollingScmProxy: run duration exceeded desired period. Duration: 19230 ms. Desired period: 1000 ms.

8:36:22.253 PM	INFO	AggregatingTimeSeriesStore Run took PT33.916S which is over the slow run threshold of PT30S. 15690 metrics written for 28 entities. PT27.952S write time over 2 writes. Longest writes: PT27.876S,PT0.076S.

8:37:02.720 PM	INFO LDBPartitionManager Updating partition=LDBPartitionMetadataWrapper{tableName=stream, partitionName=stream_2020-12-10T17:48:01.580Z, startTime=2020-12-10T17:48:01.580Z, endTime=null, version=2, state=OPEN}. Setting endTime=2020-12-10T18:38:02.709Z

Metrics Aggregation Run Duration Thresholds  

critical:30000.0,      warning:10000.0


present we are using default values in my cluster. any changes required for this values. else any other issue is there in my cluster.

[#2]

@Raj77 Your Service Monitor is running out of Java Heap hence the issue:

JvmPauseMonitor	Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 18430ms: no GCs detected.

So please increase the Java heap size for SMON to resolve the issue. Refer the Requirements here for Heap size: https://docs.cloudera.com/cloudera-manager/7.2.1/managing-clusters/topics/cm-configuring-memory-allo...

[#3]

My cluster Java Heap memory values kindly please check this values.

 Raj@77  Your Service Monitor is running out of Java Heap hence the issue:



where can i found exact location of service monitor using java heap memory value. is there any specific requirement for service monitor Java Heap value.

Is there any issue.

 /etc/default/cloudera-scm-server


 Java Options.
#
# Default value sets Java maximum heap size to 2GB, and Java maximum permanent
# generation size to 256MB.
#

export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

 Heap size: https://docs.cloudera.com/cloudera-manager/7.2.1/managing-clusters/topics/cm-configuring-memory-allo...    Link is not working

[#4 · Accepted solution]

@Raj77 You have to go CM > Cloudera Management Services > Configuration > Then Form category filter select Service Monitor 

Then you can just search for Heap size and you will get SMON heap size there. 

Pasting The Link again: https://docs.cloudera.com/cloudera-manager/7.2.1/managing-clusters/topics/cm-configuring-memory-allo...