Support Questions

Find answers, ask questions, and share your expertise

Error with memory

avatar
Explorer

How to solve this problem?

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (gcTaskThread.cpp:48), pid=2670, tid=0x00007f4421eac700
#
# JRE version: (8.0_102-b14) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- T H R E A D ---------------

Current thread (0x00007f441c019000): JavaThread "Unknown thread" [_thread_in_vm, id=2703, stack(0x00007f4421dac000,0x00007f4421ead000)]

Stack: [0x00007f4421dac000,0x00007f4421ead000], sp=0x00007f4421eab550, free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xac52aa] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4fc29b] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x5d6c6f] GCTaskThread::GCTaskThread(GCTaskManager*, unsigned int, unsigned int)+0x15f
V [libjvm.so+0x5d5a2b] GCTaskManager::initialize()+0x3ab
V [libjvm.so+0x945b2d] ParallelScavengeHeap::initialize()+0x34d
V [libjvm.so+0xa8d783] Universe::initialize_heap()+0xf3
V [libjvm.so+0xa8dcee] universe_init()+0x3e
V [libjvm.so+0x63c705] init_globals()+0x65
V [libjvm.so+0xa713ee] Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e
V [libjvm.so+0x6d0f84] JNI_CreateJavaVM+0x74
C [libjli.so+0x745e] JavaMain+0x9e
C [libpthread.so.0+0x7aa1]


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007f441c019000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=2703, stack(0x00007f4421dac000,0x00007f4421ead000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 fd:00 2373379 /usr/java/jdk1.8.0_102/bin/java
00600000-00601000 rw-p 00000000 fd:00 2373379 /usr/java/jdk1.8.0_102/bin/java
0209b000-020bc000 rw-p 00000000 00:00 0 [heap]
c1800000-d1300000 rw-p 00000000 00:00 0
d1300000-eb300000 ---p 00000000 00:00 0
eb300000-f3000000 rw-p 00000000 00:00 0
f3000000-100000000 ---p 00000000 00:00 0
368fa00000-368fa20000 r-xp 00000000 fd:00 2621442 /lib64/ld-2.12.so
368fc1f000-368fc21000 r--p 0001f000 fd:00 2621442 /lib64/ld-2.12.so
368fc21000-368fc22000 rw-p 00021000 fd:00 2621442 /lib64/ld-2.12.so
368fc22000-368fc23000 rw-p 00000000 00:00 0
368fe00000-368ff8a000 r-xp 00000000 fd:00 2621445 /lib64/libc-2.12.so
368ff8a000-369018a000 ---p 0018a000 fd:00 2621445 /lib64/libc-2.12.so
369018a000-369018e000 r--p 0018a000 fd:00 2621445 /lib64/libc-2.12.so
369018e000-3690190000 rw-p 0018e000 fd:00 2621445 /lib64/libc-2.12.so
3690190000-3690194000 rw-p 00000000 00:00 0
3690200000-3690217000 r-xp 00000000 fd:00 2621453 /lib64/libpthread-2.12.so
3690217000-3690417000 ---p 00017000 fd:00 2621453 /lib64/libpthread-2.12.so
3690417000-3690418000 r--p 00017000 fd:00 2621453 /lib64/libpthread-2.12.so
3690418000-3690419000 rw-p 00018000 fd:00 2621453 /lib64/libpthread-2.12.so
3690419000-369041d000 rw-p 00000000 00:00 0
3690600000-3690602000 r-xp 00000000 fd:00 2621608 /lib64/libdl-2.12.so
3690602000-3690802000 ---p 00002000 fd:00 2621608 /lib64/libdl-2.12.so
3690802000-3690803000 r--p 00002000 fd:00 2621608 /lib64/libdl-2.12.so
3690803000-3690804000 rw-p 00003000 fd:00 2621608 /lib64/libdl-2.12.so
3690a00000-3690a07000 r-xp 00000000 fd:00 2621457 /lib64/librt-2.12.so
3690a07000-3690c06000 ---p 00007000 fd:00 2621457 /lib64/librt-2.12.so
3690c06000-3690c07000 r--p 00006000 fd:00 2621457 /lib64/librt-2.12.so
3690c07000-3690c08000 rw-p 00007000 fd:00 2621457 /lib64/librt-2.12.so
3691200000-3691283000 r-xp 00000000 fd:00 2621578 /lib64/libm-2.12.so
3691283000-3691482000 ---p 00083000 fd:00 2621578 /lib64/libm-2.12.so
3691482000-3691483000 r--p 00082000 fd:00 2621578 /lib64/libm-2.12.so
3691483000-3691484000 rw-p 00083000 fd:00 2621578 /lib64/libm-2.12.so
7f440d000000-7f440d270000 rwxp 00000000 00:00 0
7f440d270000-7f441c000000 ---p 00000000 00:00 0
7f441c000000-7f441c04d000 rw-p 00000000 00:00 0
7f441c04d000-7f4420000000 ---p 00000000 00:00 0
7f44207a9000-7f4420a30000 rw-p 00000000 00:00 0
7f4420a30000-7f4420a31000 ---p 00000000 00:00 0
7f4420a31000-7f4420b31000 rw-p 00000000 00:00 0
7f4420b31000-7f4420b32000 ---p 00000000 00:00 0
7f4420b32000-7f4420c32000 rw-p 00000000 00:00 0
7f4420c32000-7f4420c33000 ---p 00000000 00:00 0
7f4420c33000-7f4420d33000 rw-p 00000000 00:00 0
7f4420d33000-7f4420d34000 ---p 00000000 00:00 0
7f4420d34000-7f4420eb2000 rw-p 00000000 00:00 0
7f4420eb2000-7f4420f82000 ---p 00000000 00:00 0
7f4420f82000-7f4421000000 rw-p 00000000 00:00 0
7f4421000000-7f44210cf000 ---p 00000000 00:00 0
7f44210cf000-7f442110e000 rw-p 00000000 00:00 0
7f442110e000-7f4421176000 ---p 00000000 00:00 0
7f4421176000-7f4421181000 rw-p 00000000 00:00 0
7f4421181000-7f4421537000 ---p 00000000 00:00 0
7f4421537000-7f4421551000 r-xp 00000000 fd:00 2375962 /usr/java/jdk1.8.0_102/jre/lib/amd64/libzip.so
7f4421551000-7f4421751000 ---p 0001a000 fd:00 2375962 /usr/java/jdk1.8.0_102/jre/lib/amd64/libzip.so
7f4421751000-7f4421752000 rw-p 0001a000 fd:00 2375962 /usr/java/jdk1.8.0_102/jre/lib/amd64/libzip.so
7f4421752000-7f442175f000 r-xp 00000000 fd:00 2621508 /lib64/libnss_files-2.12.so
7f442175f000-7f442195e000 ---p 0000d000 fd:00 2621508 /lib64/libnss_files-2.12.so
7f442195e000-7f442195f000 r--p 0000c000 fd:00 2621508 /lib64/libnss_files-2.12.so
7f442195f000-7f4421960000 rw-p 0000d000 fd:00 2621508 /lib64/libnss_files-2.12.so
7f4421971000-7f442199b000 r-xp 00000000 fd:00 2373849 /usr/java/jdk1.8.0_102/jre/lib/amd64/libjava.so
7f442199b000-7f4421b9b000 ---p 0002a000 fd:00 2373849 /usr/java/jdk1.8.0_102/jre/lib/amd64/libjava.so
7f4421b9b000-7f4421b9d000 rw-p 0002a000 fd:00 2373849 /usr/java/jdk1.8.0_102/jre/lib/amd64/libjava.so
7f4421b9d000-7f4421baa000 r-xp 00000000 fd:00 2375961 /usr/java/jdk1.8.0_102/jre/lib/amd64/libverify.so
7f4421baa000-7f4421daa000 ---p 0000d000 fd:00 2375961 /usr/java/jdk1.8.0_102/jre/lib/amd64/libverify.so
7f4421daa000-7f4421dac000 rw-p 0000d000 fd:00 2375961 /usr/java/jdk1.8.0_102/jre/lib/amd64/libverify.so
7f4421dac000-7f4421daf000 ---p 00000000 00:00 0
7f4421daf000-7f4421ead000 rw-p 00000000 00:00 0
7f4421ead000-7f4422b79000 r-xp 00000000 fd:00 2376175 /usr/java/jdk1.8.0_102/jre/lib/amd64/server/libjvm.so
7f4422b79000-7f4422d79000 ---p 00ccc000 fd:00 2376175 /usr/java/jdk1.8.0_102/jre/lib/amd64/server/libjvm.so
7f4422d79000-7f4422e52000 rw-p 00ccc000 fd:00 2376175 /usr/java/jdk1.8.0_102/jre/lib/amd64/server/libjvm.so
7f4422e52000-7f4422ea0000 rw-p 00000000 00:00 0
7f4422ea0000-7f4422eb5000 r-xp 00000000 fd:00 2376509 /usr/java/jdk1.8.0_102/lib/amd64/jli/libjli.so
7f4422eb5000-7f44230b5000 ---p 00015000 fd:00 2376509 /usr/java/jdk1.8.0_102/lib/amd64/jli/libjli.so
7f44230b5000-7f44230b6000 rw-p 00015000 fd:00 2376509 /usr/java/jdk1.8.0_102/lib/amd64/jli/libjli.so
7f44230b6000-7f44230b7000 rw-p 00000000 00:00 0
7f44230bb000-7f44230be000 rw-p 00000000 00:00 0
7f44230be000-7f44230c6000 rw-s 00000000 fd:00 1576232 /tmp/hsperfdata_hdfs/2670
7f44230c6000-7f44230c7000 rw-p 00000000 00:00 0
7f44230c7000-7f44230c8000 r--p 00000000 00:00 0
7f44230c8000-7f44230c9000 rw-p 00000000 00:00 0
7ffe439dc000-7ffe439f1000 rw-p 00000000 00:00 0 [stack]
7ffe439f3000-7ffe439f4000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

VM Arguments:
jvm_args: -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender
java_command: org.apache.hadoop.util.RunJar /home/progr/oracle_export/sqoop.jar
java_class_path (initial): /etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jsp-api-2.1.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/commons-beanutils-1.9.2.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jaxb-api-2.2.2.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/logredactor-1.0.3.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jackson-jaxrs-1.8.8.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/jackson-xc-1.8.8.jar:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/slf4j-log4j12.jar:/opt/c
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_102
CLASSPATH=/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*
PATH=/usr/bin:/bin
LD_LIBRARY_PATH=:/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native
SHELL=/bin/sh

Signal Handlers:
SIGSEGV: [libjvm.so+0xac5b30], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xac5b30], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x91f470], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x91f470], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x91f470], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x91f470], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x920aa0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


--------------- S Y S T E M ---------------

OS:Red Hat Enterprise Linux Server release 6.9 (Santiago)

uname:Linux 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Mar 21 12:19:18 EDT 2017 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
load average:0.00 0.01 0.00

/proc/meminfo:
MemTotal: 24591972 kB
MemFree: 10750560 kB
Buffers: 34016 kB
Cached: 530876 kB
SwapCached: 0 kB
Active: 12748112 kB
Inactive: 415840 kB
Active(anon): 12614604 kB
Inactive(anon): 190440 kB
Active(file): 133508 kB
Inactive(file): 225400 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 12369916 kB
SwapFree: 12369916 kB
Dirty: 3328 kB
Writeback: 0 kB
AnonPages: 12597328 kB
Mapped: 437352 kB
Shmem: 205988 kB
Slab: 203800 kB
SReclaimable: 76184 kB
SUnreclaim: 127616 kB
KernelStack: 48016 kB
PageTables: 192168 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24665900 kB
Committed_AS: 35445476 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 144756 kB
VmallocChunk: 34346901500 kB
HardwareCorrupted: 0 kB
AnonHugePages: 9701376 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8368 kB
DirectMap2M: 25147392 kB


CPU:total 16 (4 cores per cpu, 2 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht, tsc, tscinvbit, tscinv

/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 2
cpu cores : 4
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 1
cpu cores : 4
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 3
cpu cores : 4
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 0
cpu cores : 4
apicid : 17
initial apicid : 17
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 2
cpu cores : 4
apicid : 21
initial apicid : 21
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 12
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 13
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 1
cpu cores : 4
apicid : 19
initial apicid : 19
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 14
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.10
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5560 @ 2.80GHz
stepping : 5
microcode : 25
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 8
core id : 3
cpu cores : 4
apicid : 23
initial apicid : 23
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5599.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

 

Memory: 4k page, physical 24591972k(10750560k free), swap 12369916k(12369916k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.102-b14) for linux-amd64 JRE (1.8.0_102-b14), built on Jun 22 2016 18:43:17 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Mon Jul 10 13:32:01 2017
elapsed time: 0 seconds (0d 0h 0m 0s)

 

1 ACCEPTED SOLUTION

avatar
Contributor

that means you have many services running on your server so your server doesn't provide the memory request to Java.

I don't know your clusters setup, but it is possible many services are running that specific server.

Try to reduce the running services.

View solution in original post

11 REPLIES 11

avatar
Master Guru

@AllenGosta,

 

What you have provided is the hs_err log from a JVM crash.

It shows that the JVM essentially wanted more memory and the OS could not grant it:

 

# Out of Memory Error (gcTaskThread.cpp:48), pid=2670, tid=0x00007f4421eac700

 

Without context, it is hard to say, but I'd start with the basics and make sure that you have not overcommitted memory on the host, look at Cloudera Manager's charts for the host where the problem happened (to view memory usage, etc.)

 

Also, consider what was occurring at the time of the crash... were you doing something in particular? Where queries or jobs running?

 

Do some web searches for this crash to find suggestions, too.

These crashes are not specific to Cloudera Manager, CDH, etc.; rather, it is JVM crash information.

 

Regards,

 

Ben

avatar
Champion

What user are you loggin in .

Looks like it is unable to write core dump files based on the log files that you had shared.

 

which operating system are you runining plus your cdh /cm version.?

 

 

you can check the ulimit by firing the below command .

 

 

Login as the user you want . 

type in the terminal as 
ulimit -a 

share the values if you can 
look for the core file

usually i will set it as unlimited to the core-files.

ulimit -c unlimited

 

refernce - https://access.redhat.com/solutions/61334

 

 

avatar
Champion
@csguna

OS and kernel version from the dump.

OS:Red Hat Enterprise Linux Server release 6.9 (Santiago)
uname:Linux 2.6.32-696.1.1.el6.x86_64 #1 SMP Tue Mar 21 12:19:18 EDT 2017 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
load average:0.00 0.01 0.00
/proc/meminfo:
MemTotal: 24591972 kB
MemFree: 10750560 kB

The load average and memory usage for the system is not high.  The proc and file limit are the defaults but for a Sqoop job this may be ok.

VM Arguments:
jvm_args: -Xmx1000m -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.11.1-1.cdh5.11.1.p0.4/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender
java_command: org.apache.hadoop.util.RunJar /home/progr/oracle_export/sqoop.jar

It is running with just shy of 1 GB of heap. So it may simply be that. You can try some of the other recommendation it has.

Did this run previously in a different version of CDH or is this a new job?

avatar
Explorer

I am running java programs(sqoop import from oracle database to hdfs,sqoop export from hdfs to oracle database) by user hdfs

CDH version - CDH 5.11.1

 

avatar
Explorer

This is new job

avatar
Champion
Start by increasing the heap for the sqoop job JVM.

avatar
Explorer
Ok, thanks

avatar
Explorer
Should I increase the number of muppers?

avatar
Champion

@mbigelowmy bad I should pay attention to the posted dump .

 

@AllenGostayou can  increase the degree of parallelism but be careful while you setting up  the num-of mappers

 number of mappers should not  impact to other requests that your database needs to serve.