Member since
09-25-2018
99
Posts
6
Kudos Received
5
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1940 | 11-03-2021 02:55 AM | |
| 1564 | 09-21-2020 10:04 PM | |
| 2788 | 08-14-2020 03:20 AM | |
| 3837 | 08-20-2019 11:07 PM | |
| 9395 | 01-06-2019 07:32 PM |
11-01-2019
12:56 AM
@gsthina Thanks for your reply, post clean-up container logs are at comfortable at 660M. @EricL path for container logs are (/data/yarn/container-logs), since I have you guys with me on this I would also like to check something about yarn filecache. I see it’s consuming around 9.3G of space any way we can have this reduced? via an automated way? Following are the Yarn settings which I, think controls filecache…. However, you guys are the experts yarn.nodemanager.localizer.cache.target-size-mb = 10GB yarn.nodemanager.localizer.cache.cleanup.interval-ms = 10 Minutes Spce Used: du -shc /data/yarn/nm/usercache/MyApplication/* 981M /data/yarn/nm/usercache/ MyApplication /appcache 9.3G /data/yarn/nm/usercache/ MyApplication /filecache
... View more
10-31-2019
10:16 AM
Hi I am running out of space on one of our DN, and found that Yarn container logs are consuming around 50GB, though there are 3 Spark jobs/applications running which can be seen under Yarn > Applications tab, but I see a lot of other application ids also listed, would like to know if these could be deleted to reclaim space (the ones which are not running) Secondly, any precautions to be taken into consideration while deleting these container logs. Any guidance would be of great help. Regards Wert
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
09-10-2019
10:05 PM
@EricL When I click on File Browser from HDFS Status page, ideally it should take me to File Browser page, but here it stays at HDFS Status page. I have not come across any error in Reports Manager, have attached a snippet from RM logs. Regards Wert 2019-09-11 16:40:53,660 INFO org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer: The ratio of extra space for indexing is set with value 0.2 2019-09-11 16:40:53,660 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build 2019-09-11 16:43:08,426 INFO com.cloudera.cmf.BasicScmProxy: Failed request to SCM: 302 2019-09-11 16:43:09,426 INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. 2019-09-11 16:43:09,597 INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM 2019-09-11 16:43:09,631 INFO com.cloudera.cmf.BasicScmProxy: Authenticated to SCM. 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Wrote '/var/lib/cloudera-scm-headlamp/hdfs/aggregates.tmp' (25243 bytes) 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.LuceneImageVisitor: Finished publishing HBase metrics 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampIndex: Temporarily shutting down search service 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Saving existing quotas for later merge 2019-09-11 16:46:49,091 INFO com.cloudera.headlamp.HeadlampServiceImpl: Getting all files with quotas of type ALL 2019-09-11 16:46:49,092 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for +(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *}) 2019-09-11 16:46:49,093 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for '+(DSQUOTA:[0 TO *} NSQUOTA:[0 TO *})' took 1ms 2019-09-11 16:46:49,094 INFO com.cloudera.headlamp.HeadlampIndex: Swapping index and FS image files 2019-09-11 16:46:49,095 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old FS Image (/var/lib/cloudera-scm-headlamp/hdfs/fsimage) 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Moving new FS Image in place. 2019-09-11 16:46:49,205 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Index (/var/lib/cloudera-scm-headlamp/hdfs/index) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Index in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Deleting old Aggregate Summary (/var/lib/cloudera-scm-headlamp/hdfs/aggregates) 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Moving new Aggregate Summary in place. 2019-09-11 16:46:50,018 INFO com.cloudera.headlamp.HeadlampIndex: Starting up search service 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 0ms 2019-09-11 16:46:50,046 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating DSQUOTA in quota cache for '/' from -1 to -1 2019-09-11 16:46:53,165 INFO com.cloudera.headlamp.HeadlampServiceImpl: Performing query: PATH:/ sortBy: null sortRerverse: false limit: 10000 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Got 1 results for PATH:/ 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Search for 'PATH:/' took 2ms 2019-09-11 16:46:53,167 INFO com.cloudera.headlamp.HeadlampServiceImpl: Updating NSQUOTA in quota cache for '/' from 9223372036854775807 to 9223372036854775807 2019-09-11 16:46:53,383 INFO com.cloudera.headlamp.HeadlampIndexManager: Finished indexing HDFS services
... View more
09-09-2019
10:46 PM
Hello,
We recently upgraded from CM 5.14.x to 5.16. I have noticed that post upgraded HDFS file browser options is not working, When I go to HDFS → File Browser option there is no change , nor there is any error, I have also tried it in IE and see the same behavior.
Any help / guidance in fixing this issue.
CM & CDH Ver - 5.16.2
Regards
Wert
... View more
Labels:
- Labels:
-
Cloudera Manager
08-20-2019
11:07 PM
Hi Ben, I managed to resolve this by dropping the existing oozie database and removing oozie folder from /var/lib/, re-create oozie database and re-install oozie. Thanks wert
... View more
08-19-2019
09:57 PM
Hi Bgooley, Thanks for your reply, I get validation error on Oozie service subtab in Hue when I try to enter anything and save it. CM Version = 5.16.1 CDH Version = 5.16.2 Is there any other way this could be fixed Thanks
... View more
08-18-2019
08:49 PM
Hi, I am unable to start node manager on a node and get the attached error, any help is resolving this is much appreciated. Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:181)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:245)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/000042.sst
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.openDatabase(NMLeveldbStateStoreService.java:950)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:937)
at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:210)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 5 more
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
08-16-2019
12:10 AM
Hello All,
Due to memory issues we had to remove some components from our test cluster and one of them was oozie. Currently I am getting an error in Hue "Missing required value: Oozie Service", would appreciate any help/guide in getting this resolved.
... View more
Labels:
- Labels:
-
Apache Oozie
-
Cloudera Hue
07-26-2019
04:25 AM
Hello All, I am trying to deploy a 6-node cluster and one of the nodes I am getting attached error while activating parcels. I had tried to copy the mentioned contents from another host where the parcels are activated but I run out of space on the host where issue persists, surprisingly both the host have the same disk space of 5GB in /opt. Requesting your assistance / guidance in resolving this issue.
... View more
Labels:
- Labels:
-
Cloudera Manager
07-09-2019
11:54 PM
Hello,
We have a 3 node cluster, on one our node we see Impala Daemon in concerning state with error as (Impala Daemon Ready Check - The readiness of the Impala Daemon to process queries is not known.), while the other daemon works fine. Restarting this daemon fixes the issue momentarily, but the issue soon comes back. Attached is part of the error log.
E0710 15:36:17.214027 28015 impala-server.cc:1454] There was an error processing the impalad catalog update. Requesting a full topic update to recover: NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
I0710 15:36:18.856057 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33567
I0710 15:36:20.856436 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33568
I0710 15:36:22.856806 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33569
I0710 15:36:24.857154 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33570
I0710 15:36:25.126052 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 140.06 MB
I0710 15:36:25.791110 28015 impala-server.cc:1380] Received large catalog update(>100mb): TABLE:XXXXX is 110.30 MB
I0710 15:36:26.857511 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33571
I0710 15:36:28.857841 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33572
I0710 15:36:30.858582 27815 Frontend.java:874] Waiting for local catalog to be initialized, attempt: 33573
I0710 15:36:31.781077 28015 jni-util.cc:211] java.lang.NoClassDefFoundError: org/antlr/runtime/CharStream
at org.apache.impala.analysis.SlotRef.<init>(SlotRef.java:47)
at org.apache.impala.analysis.CUP$SqlParser$actions.case588(SqlParser.java:14267)
at org.apache.impala.analysis.CUP$SqlParser$actions.CUP$SqlParser$do_action(SqlParser.java:3909)
at org.apache.impala.analysis.SqlParser.do_action(SqlParser.java:1309)
at java_cup.runtime.lr_parser.parse(lr_parser.java:587)
at org.apache.impala.catalog.View.init(View.java:150)
at org.apache.impala.catalog.View.loadFromThrift(View.java:131)
at org.apache.impala.catalog.Table.fromThrift(Table.java:271)
at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:398)
at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:282)
at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:169)
at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:227)
at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:191)
Caused by: java.lang.ClassNotFoundException: org.antlr.runtime.CharStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
I0710 15:36:31.816437 28015 status.cc:125] NoClassDefFoundError: org/antlr/runtime/CharStream
CAUSED BY: ClassNotFoundException: org.antlr.runtime.CharStream
@ 0x95b1e9 impala::Status::Status()
@ 0xca2552 impala::JniUtil::GetJniExceptionMsg()
@ 0xba2b28 impala::Frontend::UpdateCatalogCache()
@ 0xbc0469 impala::ImpalaServer::CatalogUpdateCallback()
@ 0xc613e3 impala::StatestoreSubscriber::UpdateState()
@ 0xc670d3 impala::StatestoreSubscriberThriftIf::UpdateState()
@ 0x10f2ea8 impala::StatestoreSubscriberProcessor::process_UpdateState()
@ 0x10f20e4 impala::StatestoreSubscriberProcessor::dispatchCall()
@ 0x92b8bc apache::thrift::TDispatchProcessor::process()
@ 0xafbc4f apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0xaf6545 impala::ThriftThread::RunRunnable()
@ 0xaf7322 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0xd15193 impala::Thread::SuperviseThread()
@ 0xd158d4 boost::detail::thread_data<>::run()
@ 0x12919aa (unknown)
@ 0x7f44d68a9dd5 start_thread
@ 0x7f44d65d2ead __clone
Any help / guidance is much appericated
Regards
Wert
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager