Support Questions
Find answers, ask questions, and share your expertise

disable core dumps. spark\yarn

disable core dumps. spark\yarn

Explorer

how i can disable all cores generated by applications? 
we are based on spark\yarn

6 REPLIES 6

Re: disable core dumps. spark\yarn

@ilia987 In the service's configuration tab you should be able to see the Enable Core Dump property. You just have to uncheck and restart services if needed. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: disable core dumps. spark\yarn

Explorer

Where exactly? i am using versions::

 

HDFS 3.1.1
YARN 3.1.1
MapReduce2 3.1.1
Tez 0.9.1
Hive 3.1.0
HBase 2.0.2
Pig 0.16.0
Sqoop 1.4.7
Oozie 4.3.1
ZooKeeper 3.4.6
Storm 1.2.1
Accumulo 1.7.0
Infra Solr 0.1.0
Ambari Metrics 0.1.0
Atlas 1.1.0
Kafka 2.0.0
Knox 1.0.0
Log Search 0.5.0
Ranger 1.2.0
Ranger KMS 1.2.0
SmartSense 1.5.1.2.7.3.0-139
Spark2 2.3.2
Zeppelin Notebook 0.8.0
Druid 0.12.1
Kerberos 1.10.3-30
Superset 0.23.0

 

Re: disable core dumps. spark\yarn

@ilia987 Are you using Ambari to monitor the cluster? If yes you can refer this :

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-collect-debug-heap-dump-linux

 

If you are using Cloudera Manager then you can also goto CM > Service > Configuration tab. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: disable core dumps. spark\yarn

Explorer

we are using ambari, but it is not what i wanted.
our hosts are ubuntu 18.04.
i tried the following:

  • set ulimit -c 0,
  • set disable core dump via code for our application:

 

#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
   
void spc_limit_core(void) {
  struct rlimit rlim;
   
  rlim.rlim_cur = rlim.rlim_max = 0;
  setrlimit(RLIMIT_CORE, &rlim);
}​

 

  • but when there is an error,( out of memory) an assert is generated and then a coredump,


     

Re: disable core dumps. spark\yarn

Explorer

anyone got an idea how i can disable core dumps?

Re: disable core dumps. spark\yarn

Explorer

bumping again,

 

still trying to disable  all application core dump that run inside the containers