Reply
Explorer
Posts: 18
Registered: ‎11-11-2014
Accepted Solution

Service Monitor Timeout

Since upgrading to 5.2 we occasionally receive:

 

"Request to the Service Monitor failed. This may cause slow page responses"

 

And the logs show the following:

 

com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3687ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3876ms

 

 

Happens pretty frequently. 

 

Anyone else experiencing this / have any ideas? Never had this issue until 5.2 upgrade.

Cloudera Employee
Posts: 229
Registered: ‎09-23-2013

Re: Service Monitor Timeout

Increase heap for the Service monitor and restart the service would be the first thing to try.

Explorer
Posts: 18
Registered: ‎11-11-2014

Re: Service Monitor Timeout

Thanks had tried that a few increments, but it didn't seem to help. Upped it up again, and now things do appear better.

 

CDH 5.1 we had it set to 1GB, never had this issue.

 

CDH 5.2 it's now set to 4GB, and so far it's looking OK. 1-3GB exhibited the issue.

 

4X the memory between versions seems high but hopefully that keeps it happy.

 

Thanks

New Contributor
Posts: 4
Registered: ‎02-20-2015

Re: Service Monitor Timeout

What steps to take, if there is no change in the logs reported even after the heap size for service monitor is increased..?

Explorer
Posts: 9
Registered: ‎12-22-2014

Re: Service Monitor Timeout

In my case, i am not even able to restart the service monitor service, as due to GC not detected. so unable to reflect changes to increase heap size of service monitor... Any other suggestion ?
New Contributor
Posts: 2
Registered: ‎08-02-2016

Re: Service Monitor Timeout

worked for me, thanks!

Highlighted
Explorer
Posts: 17
Registered: ‎11-08-2018

Re: Service Monitor was failing

Hello,

 

Service monitor was failing with below error messages and i can see below warninings in service monitor logs,

 

ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper,

 

 

WARN com.cloudera.enterprise.debug.JvmPauseMonitor
Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 21465ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=21606ms

 

Can some one please respond back.

 

Thanks,

Vinod

Explorer
Posts: 17
Registered: ‎11-08-2018

Re: Service Monitor was failing

Hello Can any one please respond on this issue,

i am seeing frequent service monitor failures and getting it paused for some time and then it comes online.

 

Thanks in advance....!

Vinod

Cloudera Employee
Posts: 232
Registered: ‎01-15-2015

Re: Service Monitor was failing

Frequent pauses in the JVM by the Garbage Collector indicate the heap memory settings are too low.

 

Please increase the CM -> Cloudera Management Service -> Configuration -> Scope: Service Monitor -> Category: Resource Management -> Java Heap Size of Service Monitor in Bytes and Java Heap Size of Service Monitor in Bytes configuration property values accordingly.

 

The Service Monitor Requirements documentation chapter has guidance for the values to choose.