Cloudera Community
wert_1311
user rank
Expert Contributor
My Accepted Solutions
Show All

Service Monitor Issue

by Expert Contributor in Support Questions
‎05-22-2019 10:17 PM
‎05-22-2019 10:17 PM
Our Service Monitor has been acting weird for the last 3-4 days it keeps failing and throws random error-   Pause duration error - Average time spent paused was 22.4 second(s) (37.38%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.   Following is the error log of service monitor :   4:56:04.002 PM ERROR SafeAvroResponderServlet Error procesing Avro request org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 21 more 4:56:04.004 PM WARN log Committed before 500 Error processing POST request. Check the system logs for more information. 4:56:04.004 PM ERROR log / java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 4:56:09.003 PM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3512ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3509ms   I tried to increase Java Heap Size to 1.5Gb but it failed with out of memory error.   Kindly assist with any recommendations / guidance to fix this issue    ... View more
Labels:

Need to Change Community Nick Name

by Expert Contributor in Intros and Suggestions
‎04-17-2019 10:44 PM
‎04-17-2019 10:44 PM
  I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ?? ... View more

Re: Retention policy for Cloudera Management Servi...

by Expert Contributor in Support Questions
‎02-13-2019 10:07 PM
‎02-13-2019 10:07 PM
Hi Tony, Thanks for your reply, appericate all the hep provided by you and Gzigldrum Regards Wert ... View more

Re: Retention policy for Cloudera Management Servi...

by Expert Contributor in Support Questions
‎02-13-2019 07:39 PM
‎02-13-2019 07:39 PM
Hi Gzigldrum,   Following is the details requested, unfortunately I am unablle to run (du -hs /var/*) seems I do not have required permissions.   Host Monitor Error   Host Monitor Storage Directory Free Space: This role's Host Monitor Storage Directory is on a filesystem with less than 5.0 GiB of its space free. /var/lib/cloudera-host-monitor (free: 3.2 GiB (16.09%), capacity: 20.0 GiB) *********************************** Event Server Error    Log Directory Free Space : This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-eventserver (free: 4.9 GiB (35.25%), capacity: 14.0 GiB) *********************************** Alert Publisher Error   Log Directory Free Space : This role's Log Directory is on a filesystem with less than 10.0 GiB of its space free. /var/log/cloudera-scm-alertpublisher (free: 5.0 GiB (35.81%), capacity: 14.0 GiB             ... View more

Re: Retention policy for Cloudera Management Servi...

by Expert Contributor in Support Questions
‎02-12-2019 09:55 PM
‎02-12-2019 09:55 PM
Hello Gzigldrum,   Thanks for your reply,my issue is that health status of Host Monitor Health, Event Server Health & Alert Publisher Health shows as Bad (Red) due to directory free space issue, is there anything I can check or change to fix this ? Will deleting the logs from the location you have mentioned in the previous location help us in reducing the alerts and bring us in Green zone.   Appreciate any help in this regard.   Regards Wert ... View more

Re: Retention policy for Cloudera Management Servi...

by Expert Contributor in Support Questions
‎02-11-2019 07:30 PM
‎02-11-2019 07:30 PM
Hello Tony,   Thank you for your reply, since there is no built-in mechanism to delete logs after 7 days I guess  the best option for us, would be to adjust "firehose_time_series_storage_bytes" property.  have some questions around the same here:   Currently the Time series storage is set at 10GiB for Host Monitor & Service Monitor (screenshot attached), I presume this is the defaullt setting, can this be lowered? if yes, then will lowering this to example 5GiB get rid of the data logs and once the the status is green can we change it back to 10GiB.  As mentioend 'data is deleted only when the limit has reached', then is that limit 10Gib? will the oldest data /logs will be deleted after this threshold of 10 GiB ? Changing the parameters (as mentioned in point 1) require me to restart the Host Monitor & Service Monitor roles?       Thanks Wert ... View more

Retention policy for Cloudera Management Services

by Expert Contributor in Support Questions
‎02-10-2019 07:29 PM
‎02-10-2019 07:29 PM
Hello,   How do I change data (logs) retention policy for all Cloudera Management Services  like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days.   Cloudera Enterprise 5.11.1   Regards Wert     ... View more
Labels:

Re: Swap Memory Alerts

by Expert Contributor in Support Questions
‎01-06-2019 07:32 PM
‎01-06-2019 07:32 PM
Hi Tomas79,   Thanks for the reply, Strangely I see swapping not set to what Cloudera recommends, this is my new role seems a lot needs to be done !   Thanks  Wert. ... View more

Swap Memory Alerts

by Expert Contributor in Support Questions
‎12-30-2018 10:28 PM
‎12-30-2018 10:28 PM
Hello All,   We have being facing issues with Swap  Alerts in our Acceptance Cluster, though there are no application jobs running we still get Swap Memory Alerts. We have not configured any threshold limits for Swap Memory alerts either in Acceptance or Production Clusters (we do not see these alerts in Production Environment ).   We have checked application logs but do not see any error in them, kindly suggest if there is anything more to be checked or done.   Cluster Details:   CM - 5.11.1 CDH - 5.0 RHEL - 7.4 ... View more
Labels:

Error while moving Activity Monitor & Report Serve...

by Expert Contributor in Support Questions
‎10-15-2018 03:04 AM
‎10-15-2018 03:04 AM
Hello,   I have set a 3 node cluster for learning purposes with CM 5.11.2, due to VM resource issues, I am trying to move Activity Monitor & Report Server from Node 1 to Node 2.   While conducting this moved I initially got error " JDBC driver cannot be found. Unable to find the JDBC database jar on host : Node 2",  after going through some articles wherein it was mentioned to install 'mysql-connector-java', which I did on Node 2, post installation I am getting a different error "No database server found running on host Node 2".   Does Activity Monitor & Report Server work only on the Node where Myaql is installed, if not then how to I move these 2 services to Node 2, kindly assist / guide me in resolving this issue.   Regards Wert ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎01-17-2025 08:00 AM
Community Statistics
Member Since ‎09-25-2018 02:43 AM
Last Visited ‎01-17-2025 08:00 AM
Posts 99
Kudos received 6