Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Service Monitor Timeout

avatar
Contributor

Since upgrading to 5.2 we occasionally receive:

 

"Request to the Service Monitor failed. This may cause slow page responses"

 

And the logs show the following:

 

com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3687ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3876ms

 

 

Happens pretty frequently. 

 

Anyone else experiencing this / have any ideas? Never had this issue until 5.2 upgrade.

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Increase heap for the Service monitor and restart the service would be the first thing to try.

View solution in original post

8 REPLIES 8

avatar
Master Collaborator

Increase heap for the Service monitor and restart the service would be the first thing to try.

avatar
Contributor

Thanks had tried that a few increments, but it didn't seem to help. Upped it up again, and now things do appear better.

 

CDH 5.1 we had it set to 1GB, never had this issue.

 

CDH 5.2 it's now set to 4GB, and so far it's looking OK. 1-3GB exhibited the issue.

 

4X the memory between versions seems high but hopefully that keeps it happy.

 

Thanks

avatar
Explorer

What steps to take, if there is no change in the logs reported even after the heap size for service monitor is increased..?

avatar
Contributor
In my case, i am not even able to restart the service monitor service, as due to GC not detected. so unable to reflect changes to increase heap size of service monitor... Any other suggestion ?

avatar
New Contributor

worked for me, thanks!

avatar
Rising Star

Hello,

 

Service monitor was failing with below error messages and i can see below warninings in service monitor logs,

 

ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper,

 

 

WARN com.cloudera.enterprise.debug.JvmPauseMonitor
Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 21465ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=21606ms

 

Can some one please respond back.

 

Thanks,

Vinod

avatar
Rising Star

Hello Can any one please respond on this issue,

i am seeing frequent service monitor failures and getting it paused for some time and then it comes online.

 

Thanks in advance....!

Vinod

avatar
Super Collaborator

Frequent pauses in the JVM by the Garbage Collector indicate the heap memory settings are too low.

 

Please increase the CM -> Cloudera Management Service -> Configuration -> Scope: Service Monitor -> Category: Resource Management -> Java Heap Size of Service Monitor in Bytes and Java Heap Size of Service Monitor in Bytes configuration property values accordingly.

 

The Service Monitor Requirements documentation chapter has guidance for the values to choose.