Service Monitor Timeout

[#1]

Since upgrading to 5.2 we occasionally receive:

"Request to the Service Monitor failed. This may cause slow page responses"

And the logs show the following:

com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3687ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3876ms

Happens pretty frequently. 

Anyone else experiencing this / have any ideas? Never had this issue until 5.2 upgrade.

[#2 · Accepted solution]

Increase heap for the Service monitor and restart the service would be the first thing to try.

[#3]

Thanks had tried that a few increments, but it didn't seem to help. Upped it up again, and now things do appear better.

CDH 5.1 we had it set to 1GB, never had this issue.

CDH 5.2 it's now set to 4GB, and so far it's looking OK. 1-3GB exhibited the issue.

4X the memory between versions seems high but hopefully that keeps it happy.

Thanks

[#4]

What steps to take, if there is no change in the logs reported even after the heap size for service monitor is increased..?

[#5]

In my case, i am not even able to restart the service monitor service, as due to GC not detected. so unable to reflect changes to increase heap size of service monitor... Any other suggestion ?

[#6]

worked for me, thanks!

[#7]

Hello,

Service monitor was failing with below error messages and i can see below warninings in service monitor logs,

ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper,

WARN com.cloudera.enterprise.debug.JvmPauseMonitor
Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 21465ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=21606ms

Can some one please respond back.

Thanks,

Vinod

[#8]

Hello Can any one please respond on this issue,

i am seeing frequent service monitor failures and getting it paused for some time and then it comes online.

Thanks in advance....!

Vinod

[#9]

Frequent pauses in the JVM by the Garbage Collector indicate the heap memory settings are too low.

Please increase the CM -> Cloudera Management Service -> Configuration -> Scope: Service Monitor -> Category: Resource Management -> Java Heap Size of Service Monitor in Bytes and Java Heap Size of Service Monitor in Bytes configuration property values accordingly.

The Service Monitor Requirements documentation chapter has guidance for the values to choose.