Created 06-29-2016 02:49 PM
Hi
My Spark application was crashed and show information
LogType:stdout
Log Upload Time:Wed Jun 29 14:38:03 -0700 2016
LogLength:1096
Log Contents:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007f67baa0d221, pid=12207, tid=140083473176320
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libcaffe.so.1.0.0-rc3+0x786221] sgemm_kernel+0x21
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /yarn/nm/usercache/ubuntu/appcache/application_1467236060045_0001/container_1467236060045_0001_01_000003/hs_err_pid12207.log
but I am not able to found "/yarn/nm/usercache/ubuntu/appcache/application_1467236060045_0001/container_1467236060045_0001_01_000003/hs_err_pid12207.log" file . its deleted automatically after Spark application
finished
how to retain report file , i am running spark with yarn .
Regards
Prateek
Created 06-30-2016 01:10 AM
Try setting yarn.nodemanager.delete.debug-delay-sec to a high value, so that it doesn't clean up the files immediately. See http://spark.apache.org/docs/latest/running-on-yarn.html
Created 07-01-2016 10:33 AM
Hi thanks
yarn.nodemanager.delete.debug-delay-sec configuration solved my problem .
I am facing one more problem from last couple of months .
Problem : Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
I am running cloudera 5.7.0 in ubuntu 14.04 . and running Spark with Yarn.
I set core dump limit to unlimited in all nodes .
Edit /etc/security/limits.conf file and add belwo lines
" * soft core unlimited " line.
i rechecked using : $ ulimit -all
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 241204
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 241204
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited