Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot allocate memory fs

Cannot allocate memory fs

Contributor

I am on CDH 5.3.3 

I just created an edge node with basic client services (hive, impala, hadoop commands) 

Whenever I run hadoop fs -ls /user/hive/warehouse I get the following error

The same command works fine on my main nodes

 

 

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fbdfc900000, 22906667008, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 22906667008 bytes for committing reserved memory.
# An error report file with more information is saved as:

 

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 22906667008 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2745), pid=23877, tid=140488489957120
#
# JRE version: (7.0_67-b01) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

--------------- T H R E A D ---------------

Current thread (0x00007fc60001b800): JavaThread "Unknown thread" [_thread_in_vm, id=23926, stack(0x00007fc60679f000,0x00007fc6068a0000)]

Stack: [0x00007fc60679f000,0x00007fc6068a0000], sp=0x00007fc60689e200, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99eb8a] VMError::report_and_die()+0x2ea
V [libjvm.so+0x49721b] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x81d9ae] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
V [libjvm.so+0x81def9] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0x817caa] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0x88a7a3] PSVirtualSpace::expand_by(unsigned long)+0x53
V [libjvm.so+0x879433] PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0x103
V [libjvm.so+0x29a3b3] AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long)+0x3e3
V [libjvm.so+0x83e7e5] ParallelScavengeHeap::initialize()+0x4d5
V [libjvm.so+0x971dba] Universe::initialize_heap()+0xca
V [libjvm.so+0x972f89] universe_init()+0x79
V [libjvm.so+0x5b09f5] init_globals()+0x65
V [libjvm.so+0x95a8ed] Threads::create_vm(JavaVMInitArgs*, bool*)+0x1ed
V [libjvm.so+0x638a34] JNI_CreateJavaVM+0x74
C [libjli.so+0x2f8e] JavaMain+0x9e


--------------- P R O C E S S ---------------

Java Threads: ( => current thread )

Other Threads:

=>0x00007fc60001b800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=23926, stack(0x00007fc60679f000,0x00007fc6068a0000)]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread: None

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Internal exceptions (0 events):
No events

Events (0 events):
No events


Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 657942 /usr/lib/jvm/java-7-oracle-cloudera/bin/java
00600000-00601000 rw-p 00000000 ca:01 657942 /usr/lib/jvm/java-7-oracle-cloudera/bin/java
00b7f000-00ba0000 rw-p 00000000 00:00 0 [heap]
7fbdf36d6000-7fbdf61a9000 rw-p 00000000 00:00 0
7fbdf61a9000-7fbdf7700000 rw-p 00000000 00:00 0
7fbdf7700000-7fbdfc900000 rw-p 00000000 00:00 0
7fc351c39000-7fc5fc980000 rw-p 00000000 00:00 0
7fc5fc9cb000-7fc5fc9cc000 rw-p 00000000 00:00 0
7fc5fc9cc000-7fc5fcc3c000 rwxp 00000000 00:00 0
7fc5fcc3c000-7fc5ff9cc000 rw-p 00000000 00:00 0
7fc5ff9cc000-7fc5ff9e6000 r-xp 00000000 ca:01 657539 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libzip.so
7fc5ff9e6000-7fc5ffbe6000 ---p 0001a000 ca:01 657539 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libzip.so
7fc5ffbe6000-7fc5ffbe7000 rw-p 0001a000 ca:01 657539 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libzip.so
7fc5ffbe7000-7fc5ffbf3000 r-xp 00000000 ca:01 396905 /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc5ffbf3000-7fc5ffdf2000 ---p 0000c000 ca:01 396905 /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc5ffdf2000-7fc5ffdf3000 r--p 0000b000 ca:01 396905 /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc5ffdf3000-7fc5ffdf4000 rw-p 0000c000 ca:01 396905 /lib/x86_64-linux-gnu/libnss_files-2.15.so
7fc5ffdf4000-7fc5ffdfe000 r-xp 00000000 ca:01 396907 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc5ffdfe000-7fc5ffffe000 ---p 0000a000 ca:01 396907 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc5ffffe000-7fc5fffff000 r--p 0000a000 ca:01 396907 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc5fffff000-7fc600000000 rw-p 0000b000 ca:01 396907 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
7fc600000000-7fc600037000 rw-p 00000000 00:00 0
7fc600037000-7fc604000000 ---p 00000000 00:00 0
7fc6040bf000-7fc6040d6000 r-xp 00000000 ca:01 396902 /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc6040d6000-7fc6042d5000 ---p 00017000 ca:01 396902 /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc6042d5000-7fc6042d6000 r--p 00016000 ca:01 396902 /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc6042d6000-7fc6042d7000 rw-p 00017000 ca:01 396902 /lib/x86_64-linux-gnu/libnsl-2.15.so
7fc6042d7000-7fc6042d9000 rw-p 00000000 00:00 0
7fc6042d9000-7fc6042e1000 r-xp 00000000 ca:01 396903 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc6042e1000-7fc6044e0000 ---p 00008000 ca:01 396903 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc6044e0000-7fc6044e1000 r--p 00007000 ca:01 396903 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc6044e1000-7fc6044e2000 rw-p 00008000 ca:01 396903 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
7fc6044e2000-7fc60450b000 r-xp 00000000 ca:01 657562 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libjava.so
7fc60450b000-7fc60470b000 ---p 00029000 ca:01 657562 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libjava.so
7fc60470b000-7fc60470d000 rw-p 00029000 ca:01 657562 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libjava.so
7fc60470d000-7fc60471a000 r-xp 00000000 ca:01 657558 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libverify.so
7fc60471a000-7fc604919000 ---p 0000d000 ca:01 657558 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libverify.so
7fc604919000-7fc60491b000 rw-p 0000c000 ca:01 657558 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/libverify.so
7fc60491b000-7fc604922000 r-xp 00000000 ca:01 396923 /lib/x86_64-linux-gnu/librt-2.15.so
7fc604922000-7fc604b21000 ---p 00007000 ca:01 396923 /lib/x86_64-linux-gnu/librt-2.15.so
7fc604b21000-7fc604b22000 r--p 00006000 ca:01 396923 /lib/x86_64-linux-gnu/librt-2.15.so
7fc604b22000-7fc604b23000 rw-p 00007000 ca:01 396923 /lib/x86_64-linux-gnu/librt-2.15.so
7fc604b23000-7fc604c1e000 r-xp 00000000 ca:01 396895 /lib/x86_64-linux-gnu/libm-2.15.so
7fc604c1e000-7fc604e1d000 ---p 000fb000 ca:01 396895 /lib/x86_64-linux-gnu/libm-2.15.so
7fc604e1d000-7fc604e1e000 r--p 000fa000 ca:01 396895 /lib/x86_64-linux-gnu/libm-2.15.so
7fc604e1e000-7fc604e1f000 rw-p 000fb000 ca:01 396895 /lib/x86_64-linux-gnu/libm-2.15.so
7fc604e1f000-7fc60598f000 r-xp 00000000 ca:01 657525 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/server/libjvm.so
7fc60598f000-7fc605b8e000 ---p 00b70000 ca:01 657525 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/server/libjvm.so
7fc605b8e000-7fc605c53000 rw-p 00b6f000 ca:01 657525 /usr/lib/jvm/java-7-oracle-cloudera/jre/lib/amd64/server/libjvm.so
7fc605c53000-7fc605c93000 rw-p 00000000 00:00 0
7fc605c93000-7fc605e48000 r-xp 00000000 ca:01 396879 /lib/x86_64-linux-gnu/libc-2.15.so
7fc605e48000-7fc606048000 ---p 001b5000 ca:01 396879 /lib/x86_64-linux-gnu/libc-2.15.so
7fc606048000-7fc60604c000 r--p 001b5000 ca:01 396879 /lib/x86_64-linux-gnu/libc-2.15.so
7fc60604c000-7fc60604e000 rw-p 001b9000 ca:01 396879 /lib/x86_64-linux-gnu/libc-2.15.so
7fc60604e000-7fc606053000 rw-p 00000000 00:00 0
7fc606053000-7fc606055000 r-xp 00000000 ca:01 396885 /lib/x86_64-linux-gnu/libdl-2.15.so
7fc606055000-7fc606255000 ---p 00002000 ca:01 396885 /lib/x86_64-linux-gnu/libdl-2.15.so
7fc606255000-7fc606256000 r--p 00002000 ca:01 396885 /lib/x86_64-linux-gnu/libdl-2.15.so
7fc606256000-7fc606257000 rw-p 00003000 ca:01 396885 /lib/x86_64-linux-gnu/libdl-2.15.so
7fc606257000-7fc60626d000 r-xp 00000000 ca:01 657948 /usr/lib/jvm/java-7-oracle-cloudera/lib/amd64/jli/libjli.so
7fc60626d000-7fc60646d000 ---p 00016000 ca:01 657948 /usr/lib/jvm/java-7-oracle-cloudera/lib/amd64/jli/libjli.so
7fc60646d000-7fc60646e000 rw-p 00016000 ca:01 657948 /usr/lib/jvm/java-7-oracle-cloudera/lib/amd64/jli/libjli.so
7fc60646e000-7fc606486000 r-xp 00000000 ca:01 396920 /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc606486000-7fc606685000 ---p 00018000 ca:01 396920 /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc606685000-7fc606686000 r--p 00017000 ca:01 396920 /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc606686000-7fc606687000 rw-p 00018000 ca:01 396920 /lib/x86_64-linux-gnu/libpthread-2.15.so
7fc606687000-7fc60668b000 rw-p 00000000 00:00 0
7fc60668b000-7fc6066ad000 r-xp 00000000 ca:01 396870 /lib/x86_64-linux-gnu/ld-2.15.so
7fc6066d7000-7fc6066e1000 rw-p 00000000 00:00 0
7fc6066e1000-7fc606797000 rw-p 00000000 00:00 0
7fc606797000-7fc60679f000 rw-s 00000000 ca:01 919865 /tmp/hsperfdata_waterlinedata/23877
7fc60679f000-7fc6067a2000 ---p 00000000 00:00 0
7fc6067a2000-7fc6068a4000 rw-p 00000000 00:00 0
7fc6068a8000-7fc6068aa000 rw-p 00000000 00:00 0
7fc6068aa000-7fc6068ab000 r--p 00000000 00:00 0
7fc6068ab000-7fc6068ad000 rw-p 00000000 00:00 0
7fc6068ad000-7fc6068ae000 r--p 00022000 ca:01 396870 /lib/x86_64-linux-gnu/ld-2.15.so
7fc6068ae000-7fc6068b0000 rw-p 00023000 ca:01 396870 /lib/x86_64-linux-gnu/ld-2.15.so
7fff5a727000-7fff5a749000 rw-p 00000000 00:00 0 [stack]
7fff5a7ff000-7fff5a800000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

VM Arguments:
jvm_args: -Xmx1000m -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms34359738368 -Xmx34359738368 -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender
java_command: org.apache.hadoop.fs.FsShell -ls /user/hive/warehouse
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/lib/jvm/java-7-oracle-cloudera
CLASSPATH=/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/libexec/../../hadoop-mapreduce/.//*
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
USERNAME=waterlinedata
LD_LIBRARY_PATH=:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hadoop/lib/native
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x81c060], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000


--------------- S Y S T E M ---------------

OS:wheezy/sid

uname:Linux 3.2.0-59-virtual #90-Ubuntu SMP Tue Jan 7 23:02:33 UTC 2014 x86_64
libc:glibc 2.15 NPTL 2.15
rlimit: STACK 8192k, CORE 0k, NPROC 136509, NOFILE 4096, AS infinity
load average:0.00 0.01 0.05

/proc/meminfo:
MemTotal: 17489776 kB
MemFree: 9107976 kB
Buffers: 237164 kB
Cached: 6575156 kB
SwapCached: 0 kB
Active: 3954620 kB
Inactive: 3906560 kB
Active(anon): 1051864 kB
Inactive(anon): 836 kB
Active(file): 2902756 kB
Inactive(file): 3905724 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 44 kB
Writeback: 0 kB
AnonPages: 1048880 kB
Mapped: 51664 kB
Shmem: 3844 kB
Slab: 303320 kB
SReclaimable: 289996 kB
SUnreclaim: 13324 kB
KernelStack: 2256 kB
PageTables: 6976 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8744888 kB
Committed_AS: 13486712 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 65880 kB
VmallocChunk: 34359670376 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 17928192 kB
DirectMap2M: 0 kB


CPU:total 2 (4 cores per cpu, 2 threads per core) family 6 model 26 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht, tsc, tscinvbit, tscinv

/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
microcode : 0x11
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz
stepping : 5
microcode : 0x11
cpu MHz : 2666.760
cache size : 8192 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc pni ssse3 cx16 sse4_1 sse4_2 popcnt hypervisor lahf_lm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

 

Memory: 4k page, physical 17489776k(9107976k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (24.65-b04) for linux-amd64 JRE (1.7.0_67-b01), built on Jul 25 2014 09:20:21 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Wed Apr 15 21:32:42 2015
elapsed time: 0 seconds

1 REPLY 1
Highlighted

Re: Cannot allocate memory fs

Master Guru
> Native memory allocation (malloc) failed to allocate 22906667008 bytes for committing reserved memory.

The above suggests that your JVM is trying to perhaps start with too
high a memory request (its close to 21 GB per that number). What is
the heap set in /etc/hadoop/conf/hadoop-env.sh for HADOOP_CLIENT_OPTS
and HADOOP_OPTS?

Per the dump's VM arguments, it appears the command is passing -Xmx
and -Xms values of 32 GB, which is the likely cause. The above env
file should help you fix this, unless you've set this intentionally
(in which case my question is - Why do you need so much heap for
clients?).