Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error report file is deleted automatically after spark application finished

Error report file is deleted automatically after spark application finished

Hi

My Spark application was crashed and show information

LogType:stdout
Log Upload Time:Wed Jun 29 14:38:03 -0700 2016
LogLength:1096
Log Contents:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007f67baa0d221, pid=12207, tid=140083473176320
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libcaffe.so.1.0.0-rc3+0x786221]  sgemm_kernel+0x21
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /yarn/nm/usercache/ubuntu/appcache/application_1467236060045_0001/container_1467236060045_0001_01_000003/hs_err_pid12207.log



but I am not able to found  "/yarn/nm/usercache/ubuntu/appcache/application_1467236060045_0001/container_1467236060045_0001_01_000003/hs_err_pid12207.log"  file . its deleted  automatically after Spark application
 finished


how  to retain report file , i am running spark with yarn .

Regards
Prateek

2 REPLIES 2
Highlighted

Re: Error report file is deleted automatically after spark application finished

Master Collaborator

Try setting yarn.nodemanager.delete.debug-delay-sec to a high value, so that it doesn't clean up the files immediately. See http://spark.apache.org/docs/latest/running-on-yarn.html

Re: Error report file is deleted automatically after spark application finished

Hi thanks

 

yarn.nodemanager.delete.debug-delay-sec configuration  solved my problem .

 

I am facing one more problem from last couple of months .

 

 Problem :  Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

 

I am running cloudera 5.7.0 in ubuntu 14.04 . and running Spark with Yarn.

 

I set core dump limit to unlimited in all nodes .
   Edit  /etc/security/limits.conf file and add  belwo lines

 " * soft core unlimited " line.

i rechecked  using :  $ ulimit -all

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 241204
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 241204
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited