About wert_1311

wert_1311 · ‎09-10-2019

@EricL When I click on File Browser from HDFS Status page, ideally it should take me to File Browser page, but here it stays at HDFS Status page. I have not come across any error in Reports Manager, have attached a snippet from RM logs. Regards Wert 2019-09-11 16:40:53,660 INFO org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer: The ratio of extra space for indexing is set with value 0.2 2019-09-11 16:40:53,660 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build 2019-09-11 16:43:08,426 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-09-11 16:43:09,426 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-09-11 16:43:09,597 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-09-11 16:43:09,631 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/aggregates.tmp' (25243 bytes) 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL 2019-09-11 16:46:49,092 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms 2019-09-11 16:46:49,094 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files 2019-09-11 16:46:49,095 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/fsimage) 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place. 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/index) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/aggregates) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 0ms 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1 2019-09-11 16:46:53,165 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 2ms 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807 2019-09-11 16:46:53,383 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services

wert_1311 · ‎09-09-2019

Hello, We recently upgraded from CM 5.14.x to 5.16. I have noticed that post upgraded HDFS file browser options is not working, When I go to HDFS → File Browser option there is no change , nor there is any error, I have also tried it in IE and see the same behavior. Any help / guidance in fixing this issue. CM & CDH Ver - 5.16.2 Regards Wert

wert_1311 · ‎08-20-2019

Hi Ben, I managed to resolve this by dropping the existing oozie database and removing oozie folder from /var/lib/, re-create oozie database and re-install oozie. Thanks wert

wert_1311 · ‎08-19-2019

Hi Bgooley, Thanks for your reply, I get validation error on Oozie service subtab in Hue when I try to enter anything and save it. CM Version = 5.16.1 CDH Version = 5.16.2 Is there any other way this could be fixed Thanks

wert_1311 · ‎08-18-2019

Hi, I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated. Error starting NodeManager org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:950) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:937) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:210) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more

wert_1311 · ‎08-16-2019

Hello All, Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.

wert_1311 · ‎07-26-2019

Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.

wert_1311 · ‎07-09-2019

Hello, We have a 3 node cluster, on one our node we see Impala Daemon in concerning state with error as (Impala Daemon Ready Check - The readiness of the Impala Daemon to process queries is not known.), while the other daemon works fine. Restarting this daemon fixes the issue momentarily, but the issue soon comes back. Attached is part of the error log. E0710 15:36:17.214027 28015 impala-server.cc:1454] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NoClassDefFoundError: org/antlr/runtime/CharStream CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream I0710 15:36:18.856057 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33567 I0710 15:36:20.856436 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33568 I0710 15:36:22.856806 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33569 I0710 15:36:24.857154 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33570 I0710 15:36:25.126052 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 140.06 MB I0710 15:36:25.791110 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 110.30 MB I0710 15:36:26.857511 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33571 I0710 15:36:28.857841 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33572 I0710 15:36:30.858582 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33573 I0710 15:36:31.781077 28015 jni-util.cc:211] java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream at org.apache.impala.analysis.SlotRef.<init>(SlotRef.java:47) at org.apache.impala.analysis.CUP$SqlParser$actions.case588(SqlParser.java:14267) at org.apache.impala.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:3909) at org.apache.impala.analysis.SqlParser.do_action(SqlParser.java:1309) at java_cup.runtime.lr_parser.parse(lr_parser.java:587) at org.apache.impala.catalog.View.init(View.java:150) at org.apache.impala.catalog.View.loadFromThrift(View.java:131) at org.apache.impala.catalog.Table.fromThrift(Table.java:271) at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:398) at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:282) at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:169) at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:227) at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191) Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.CharStream at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 13 more I0710 15:36:31.816437 28015 status.cc:125] NoClassDefFoundError: org/antlr/runtime/CharStream CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream @ 0x95b1e9 impala::Status::Status() @ 0xca2552 impala::JniUtil::GetJniExceptionMsg() @ 0xba2b28 impala::Frontend::UpdateCatalogCache() @ 0xbc0469 impala::ImpalaServer::CatalogUpdateCallback() @ 0xc613e3 impala::StatestoreSubscriber::UpdateState() @ 0xc670d3 impala::StatestoreSubscriberThriftIf::UpdateState() @ 0x10f2ea8 impala::StatestoreSubscriberProcessor::process_UpdateState() @ 0x10f20e4 impala::StatestoreSubscriberProcessor::dispatchCall() @ 0x92b8bc apache::thrift::TDispatchProcessor::process() @ 0xafbc4f apache::thrift::server::TAcceptQueueServer::Task::run() @ 0xaf6545 impala::ThriftThread::RunRunnable() @ 0xaf7322 boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0xd15193 impala::Thread::SuperviseThread() @ 0xd158d4 boost::detail::thread_data<>::run() @ 0x12919aa (unknown) @ 0x7f44d68a9dd5 start_thread @ 0x7f44d65d2ead __clone Any help / guidance is much appericated Regards Wert

wert_1311 · ‎06-25-2019

I have a small POC cluster for our Dev team wherein disk space is precious. I see that a lot of data and metadata located in: data/kudu/ts/data hence was hoping if I could delete them and recover space. Since I am unsure whether deleteing this would affect application performace or render it useless, wanted some information / advise on this.

wert_1311 · ‎06-21-2019

Hello All, Would there be any implications on Kudu applications performance etc, in case we delete Kudu TS data. Requesting any help / guidance on the same. Regards Wert

wert_1311 · ‎06-17-2019

Hell All, My understanding of Canary files was that data is written to a file and once the test is completed, this file is removed automatically. I recently ran a Directory Usage report and I came across many of these files (/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2018_12_11-23_46_17). The size shows as zero KB, and the File and Directory Count =1 do these also gets stored in the Namenode Metadata? Secondly, would these get auto deleted from hdfs? If not, would deleting them manually have any repercussions on Hadoop Cluster? Are there any best practice / guide to effectively manage these files, Kindly advice. Regards Wert

wert_1311 · ‎06-02-2019

Hi Li, Thanks for your help Regards Wert

wert_1311 · ‎05-23-2019

Hello, We have been facing issues with our CM Service Monitor, for the past couple of days it fails and the starts to work again, we also get the below alerts repeatedly : Concerning : The last metrics aggregation run duration is 25.4 second(s). Warning threshold: 10 second(s). Concerning : Average time spent paused was 27.9 second(s) (46.47%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. There are no configuration changes done on this node, attached is the SM log file. 5:48:14.372 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2543ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2718ms 5:48:14.373 AM ERROR SafeAvroResponderServlet Error procesing Avro request org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 21 more 5:48:14.375 AM WARN log Committed before 500 Error processing POST request. Check the system logs for more information. 5:48:14.375 AM ERROR log / java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 5:48:32.030 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3051ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3431ms 5:48:36.128 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2597ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2962ms 5:48:39.909 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2781ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2863ms 5:48:42.965 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2049ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2480ms 5:48:46.620 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3155ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3204ms 5:48:49.569 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2448ms: GC pool 'ParNew' had collection(s): count=1 time=0ms, GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2534ms 5:48:53.360 AM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 2790ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=2807ms Any help / guidance is appreciated

wert_1311 · ‎05-22-2019

Our Service Monitor has been acting weird for the last 3-4 days it keeps failing and throws random error- Pause duration error - Average time spent paused was 22.4 second(s) (37.38%) per minute over the previous 5 minute(s). Warning threshold: 30.00%. Following is the error log of service monitor : 4:56:04.002 PM ERROR SafeAvroResponderServlet Error procesing Avro request org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:569) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:623) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeLength(SafeAvroHttpTransceiver.java:128) at com.cloudera.enterprise.SafeAvroHttpTransceiver.writeBuffers(SafeAvroHttpTransceiver.java:120) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:57) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.mortbay.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:368) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:129) at org.mortbay.io.bio.StreamEndPoint.flush(StreamEndPoint.java:161) at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:714) ... 21 more 4:56:04.004 PM WARN log Committed before 500 Error processing POST request. Check the system logs for more information. 4:56:04.004 PM ERROR log / java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1023) at org.mortbay.jetty.Response.sendError(Response.java:240) at com.cloudera.enterprise.SafeAvroResponderServlet.logAndSuppressException(SafeAvroResponderServlet.java:69) at com.cloudera.enterprise.SafeAvroResponderServlet.doPost(SafeAvroResponderServlet.java:59) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 4:56:09.003 PM INFO JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 3512ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=3509ms I tried to increase Java Heap Size to 1.5Gb but it failed with out of memory error. Kindly assist with any recommendations / guidance to fix this issue

wert_1311 · ‎05-08-2019

We have been getting swap alerts on one of our worker nodes; have checked swappiness setting on this particular worker node and, vm.swappiness is set to 1 as recommended. We see these alerts come up on Yarn, Impala and Kudu. CM – 5.14 Swap Space - 8GB Physical Memory – 65 GB OS – Redhat 7.6 Any help /guidance is appreciated.

wert_1311 · ‎04-17-2019

I need to change my Community Nickname but the option is disabled, is there a way I can change my nickname ??

wert_1311 · ‎04-17-2019

Hello All, I have been asked to create some Health Check report of our Hadoop Cluster for Senior Management, I see all the reports in Cloudera Manger are Chart based and as per my understanding one needs access to cloudera manager console to access these charts, is there a way that these dashboard charts can be access without the need to login to cloudera manager console or has anyone created some reports which can be emailed daily in an excel, PowerPoint format etc. Any help / pointers on this is much appreciated.

wert_1311 · ‎02-20-2019

Hello All, When I login to our Name Node, terminal window and run klist I get the below details, [my_id@MYHOSTNAME ~]$ klist Ticket cache: KEYRING:persistent:309057:krb_ccache_0yNoeTe Default principal: MY_ID@MY DOMAIN.COM Valid starting Expires Service principal 02/21/2019 18:31:50 02/22/2019 04:31:50 krbtgt/MY DOMAIN.COM renew until 02/28/2019 18:31:50 When I sudo -su hdfs I see that I am on hdfs as I get : "hdfs@Hostname My_ID" as soon as I try to run anyhdfs commands e.g. - hdfs dfsadmin -report, I get a long error, below is a part of it: Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 41 more ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcep tion: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "MY_HOSTNAME.MY DOMAIN.com/IP ADDRESS"; destination host is: " MY_HOSTNAME.MY DOMAIN.com":8021; Kindly suggest how do I fix this, kinda newbie with kerberos. Regards Wert

wert_1311 · ‎02-13-2019

Hi Tony, Thanks for your reply, appericate all the hep provided by you and Gzigldrum Regards Wert

wert_1311 · ‎02-13-2019

Hi Gzigldrum, Following is the details requested, unfortunately I am unablle to run (du -hs /var/*) seems I do not have required permissions. Host Monitor Error Host Monitor Storage Directory Free Space: This role's Host Monitor Storage Directory is on a filesystem with less than 5.0 GiB of its space free. /var/lib/cloudera-host-monitor (free: 3.2 GiB (16.09%), capacity: 20.0 GiB) *********************************** Event Server Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 5.0 GiB of its space free. /var/log/cloudera-scm-eventserver (free: 4.9 GiB (35.25%), capacity: 14.0 GiB) *********************************** Alert Publisher Error Log Directory Free Space : This role's Log Directory is on a filesystem with less than 10.0 GiB of its space free. /var/log/cloudera-scm-alertpublisher (free: 5.0 GiB (35.81%), capacity: 14.0 GiB

wert_1311 · ‎02-12-2019

Hello Gzigldrum, Thanks for your reply,my issue is that health status of Host Monitor Health, Event Server Health & Alert Publisher Health shows as Bad (Red) due to directory free space issue, is there anything I can check or change to fix this ? Will deleting the logs from the location you have mentioned in the previous location help us in reducing the alerts and bring us in Green zone. Appreciate any help in this regard. Regards Wert

wert_1311 · ‎02-11-2019

Hello Tony, Thank you for your reply, since there is no built-in mechanism to delete logs after 7 days I guess the best option for us, would be to adjust " firehose_time_series_storage_bytes" property. have some questions around the same here: Currently the Time series storage is set at 10GiB for Host Monitor & Service Monitor (screenshot attached) , I presume this is the defaullt setting, can this be lowered? if yes, then will lowering this to example 5GiB get rid of the data logs and once the the status is green can we change it back to 10GiB. As mentioend 'data is deleted only when the limit has reached', then is that limit 10Gib? will the oldest data /logs will be deleted after this threshold of 10 GiB ? Changing the parameters (as mentioned in point 1) require me to restart the Host Monitor & Service Monitor roles? Thanks Wert

wert_1311 · ‎02-10-2019

Hello, How do I change data (logs) retention policy for all Cloudera Management Services like Service Monitor, Host Monitor and Event Server. My target is to retain data (logs) only last 7 days. Cloudera Enterprise 5.11.1 Regards Wert

wert_1311 · ‎01-06-2019

Hi Tomas79, Thanks for the reply, Strangely I see swapping not set to what Cloudera recommends, this is my new role seems a lot needs to be done ! Thanks Wert.

wert_1311 · ‎12-30-2018

Hello All, We have being facing issues with Swap Alerts in our Acceptance Cluster, though there are no application jobs running we still get Swap Memory Alerts. We have not configured any threshold limits for Swap Memory alerts either in Acceptance or Production Clusters (we do not see these alerts in Production Environment ). We have checked application logs but do not see any error in them, kindly suggest if there is anything more to be checked or done. Cluster Details: CM - 5.11.1 CDH - 5.0 RHEL - 7.4

wert_1311 · ‎10-15-2018

Hello, I have set a 3 node cluster for learning purposes with CM 5.11.2, due to VM resource issues, I am trying to move Activity Monitor & Report Server from Node 1 to Node 2. While conducting this moved I initially got error " JDBC driver cannot be found. Unable to find the JDBC database jar on host : Node 2", after going through some articles wherein it was mentioned to install 'mysql-connector-java', which I did on Node 2, post installation I am getting a different error "No database server found running on host Node 2". Does Activity Monitor & Report Server work only on the Node where Myaql is installed, if not then how to I move these 2 services to Node 2, kindly assist / guide me in resolving this issue. Regards Wert

Online	Offline
Last Visited	‎09-26-2019 04:23 AM

Date Registered	‎09-25-2018 02:43 AM
Last Visited	‎09-26-2019 04:23 AM
Total Messages Posted	31
Total Kudos Received	1

Re: Hue Error

Re: Swap Memory Alerts

Re: HDFS File Browser not working

HDFS File Browser not working

Re: Hue Error

Re: Hue Error

Unable to start Node manager

Hue Error

Error while activating parcels

Impala daemon stuck at Ready Check

Re: Kudu TS Data

Kudu TS Data

Canary Files

Re: Service Monitor restarts repeatedly

Service Monitor restarts repeatedly

Service Monitor Issue

Swap Alerts

Need to Change Community Nick Name

Health Check Reports

Kerberos Error

Re: Retention policy for Cloudera Management Servi...

Re: Retention policy for Cloudera Management Servi...

Re: Retention policy for Cloudera Management Servi...

Re: Retention policy for Cloudera Management Servi...

Retention policy for Cloudera Management Services

Re: Swap Memory Alerts

Swap Memory Alerts

Error while moving Activity Monitor & Report Serve...