Member since
09-25-2018
99
Posts
6
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2092 | 11-03-2021 02:55 AM | |
1635 | 09-21-2020 10:04 PM | |
2927 | 08-14-2020 03:20 AM | |
3994 | 08-20-2019 11:07 PM | |
9670 | 01-06-2019 07:32 PM |
06-02-2019
09:06 PM
1 Kudo
Hi Li, Thanks for your help Regards Wert
... View more
05-23-2019
06:23 PM
Hello, We have been facing issues with our CM Service Monitor, for the past couple of days it fails and the starts to work again, we also get the below alerts repeatedly : Concerning : The last metrics aggregation run duration is 25.4 second(s). Warning threshold: 10 second(s). Concerning : Average time spent paused was 27.9 second(s) (46.47%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. There are no configuration changes done on this node, attached is the SM log file. 5:48:14.372 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2543ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2718ms
5:48:14.373 AM ERROR SafeAvroResponderServlet
Error procesing Avro request
org.mortbay.jetty.EofException
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569)
at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623)
at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128)
at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120)
at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368)
at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129)
at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161)
at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714)
... 21 more
5:48:14.375 AM WARN log Committed before 500 Error processing POST request. Check the system logs for more information.
5:48:14.375 AM ERROR log
/
java.lang.IllegalStateException: Committed
at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
at org.mortbay.jetty.Response.sendError(Response.java:240)
at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69)
at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
5:48:32.030 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3431ms
5:48:36.128 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2597ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2962ms
5:48:39.909 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2781ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2863ms
5:48:42.965 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2049ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2480ms
5:48:46.620 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3155ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3204ms
5:48:49.569 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2448ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2534ms
5:48:53.360 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2790ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2807ms
Any help / guidance is appreciated
... View more
Labels:
- Labels:
-
Cloudera Manager
05-22-2019
10:17 PM
Our Service Monitor has been acting weird for the last 3-4 days it keeps failing and throws random error- Pause duration error - Average time spent paused was 22.4 second(s) (37.38%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. Following is the error log of service monitor : 4:56:04.002 PM ERROR SafeAvroResponderServlet Error procesing Avro request org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 21 more 4:56:04.004 PM WARN log Committed before 500 Error processing POST request. Check the system logs for more information. 4:56:04.004 PM ERROR log / java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 4:56:09.003 PM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3512ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3509ms I tried to increase Java Heap Size to 1.5Gb but it failed with out of memory error. Kindly assist with any recommendations / guidance to fix this issue
... View more
Labels:
- Labels:
-
Cloudera Manager
04-17-2019
10:44 PM
I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ??
... View more
02-13-2019
10:07 PM
Hi Tony, Thanks for your reply, appericate all the hep provided by you and Gzigldrum Regards Wert
... View more
02-13-2019
07:39 PM
Hi Gzigldrum, Following is the details requested, unfortunately I am unablle to run (du -hs /var/*) seems I do not have required permissions. Host Monitor Error Host Monitor Storage Directory Free Space: This role's Host Monitor Storage Directory is on a filesystem with less than 5.0 GiB of its space free. /var/lib/cloudera-host-monitor (free: 3.2 GiB (16.09%), capacity: 20.0 GiB) *********************************** Event Server Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-eventserver (free: 4.9 GiB (35.25%), capacity: 14.0 GiB) *********************************** Alert Publisher Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 10.0 GiB of its space free. /var/log/cloudera-scm-alertpublisher (free: 5.0 GiB (35.81%), capacity: 14.0 GiB
... View more
02-12-2019
09:55 PM
Hello Gzigldrum, Thanks for your reply,my issue is that health status of Host Monitor Health, Event Server Health & Alert Publisher Health shows as Bad (Red) due to directory free space issue, is there anything I can check or change to fix this ? Will deleting the logs from the location you have mentioned in the previous location help us in reducing the alerts and bring us in Green zone. Appreciate any help in this regard. Regards Wert
... View more
02-11-2019
07:30 PM
Hello Tony, Thank you for your reply, since there is no built-in mechanism to delete logs after 7 days I guess the best option for us, would be to adjust "firehose_time_series_storage_bytes" property. have some questions around the same here: Currently the Time series storage is set at 10GiB for Host Monitor & Service Monitor (screenshot attached), I presume this is the defaullt setting, can this be lowered? if yes, then will lowering this to example 5GiB get rid of the data logs and once the the status is green can we change it back to 10GiB. As mentioend 'data is deleted only when the limit has reached', then is that limit 10Gib? will the oldest data /logs will be deleted after this threshold of 10 GiB ? Changing the parameters (as mentioned in point 1) require me to restart the Host Monitor & Service Monitor roles? Thanks Wert
... View more
02-10-2019
07:29 PM
Hello, How do I change data (logs) retention policy for all Cloudera Management Services like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days. Cloudera Enterprise 5.11.1 Regards Wert
... View more
Labels:
- Labels:
-
Cloudera Manager
01-06-2019
07:32 PM
Hi Tomas79, Thanks for the reply, Strangely I see swapping not set to what Cloudera recommends, this is my new role seems a lot needs to be done ! Thanks Wert.
... View more
- « Previous
- Next »