Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Service Monitor Timeout

Solved Go to solution

Service Monitor Timeout

Explorer

Since upgrading to 5.2 we occasionally receive:

 

"Request to the Service Monitor failed. This may cause slow page responses"

 

And the logs show the following:

 

com.cloudera.enterprise.debug.JvmPauseMonitor: Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3687ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3876ms

 

 

Happens pretty frequently. 

 

Anyone else experiencing this / have any ideas? Never had this issue until 5.2 upgrade.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Service Monitor Timeout

Super Collaborator

Increase heap for the Service monitor and restart the service would be the first thing to try.

8 REPLIES 8

Re: Service Monitor Timeout

Super Collaborator

Increase heap for the Service monitor and restart the service would be the first thing to try.

Re: Service Monitor Timeout

Explorer

Thanks had tried that a few increments, but it didn't seem to help. Upped it up again, and now things do appear better.

 

CDH 5.1 we had it set to 1GB, never had this issue.

 

CDH 5.2 it's now set to 4GB, and so far it's looking OK. 1-3GB exhibited the issue.

 

4X the memory between versions seems high but hopefully that keeps it happy.

 

Thanks

Re: Service Monitor Timeout

New Contributor

What steps to take, if there is no change in the logs reported even after the heap size for service monitor is increased..?

Re: Service Monitor Timeout

In my case, i am not even able to restart the service monitor service, as due to GC not detected. so unable to reflect changes to increase heap size of service monitor... Any other suggestion ?

Re: Service Monitor Timeout

New Contributor

worked for me, thanks!

Highlighted

Re: Service Monitor was failing

Explorer

Hello,

 

Service monitor was failing with below error messages and i can see below warninings in service monitor logs,

 

ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper,

 

 

WARN com.cloudera.enterprise.debug.JvmPauseMonitor
Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 21465ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=21606ms

 

Can some one please respond back.

 

Thanks,

Vinod

Re: Service Monitor was failing

Explorer

Hello Can any one please respond on this issue,

i am seeing frequent service monitor failures and getting it paused for some time and then it comes online.

 

Thanks in advance....!

Vinod

Re: Service Monitor was failing

Expert Contributor

Frequent pauses in the JVM by the Garbage Collector indicate the heap memory settings are too low.

 

Please increase the CM -> Cloudera Management Service -> Configuration -> Scope: Service Monitor -> Category: Resource Management -> Java Heap Size of Service Monitor in Bytes and Java Heap Size of Service Monitor in Bytes configuration property values accordingly.

 

The Service Monitor Requirements documentation chapter has guidance for the values to choose.

Don't have an account?
Coming from Hortonworks? Activate your account here