Getting out of memory (on 8 core, 16MB, 4 node cluster), with simple HDFS operations

[#1]

E.g.

hdfs dfs -put const.txt /qtests/file3

(file is <1MB)

Error is:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f3400000, 123731968, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 123731968 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/centos/hs_err_pid4314.log

...

Max Heap size on data nodes is 1GB.

[#2 · Accepted solution]

@Steve Kiaie

Do you have enough free memory available at the OS? (during crash) And there is not additional JVM options set globally?

# free -m
# echo $JAVA_OPTIONS
# echo $HADOOP_OPTS
# echo $HADOOP_CLIENT_OPTS

Also this error / crash indicates that it you might have set the system OS memory overcommit setting was 2

Please check this file "/proc/sys/vm/overcommit_memory" This switch knows 3 different settings 0 /1 /2:

0: The Linux kernel is free to overcommit memory (this is the default), a heuristic algorithm is applied to figure out if enough memory is available.
1: The Linux kernel will always overcommit memory, and never check if enough memory is available. This increases the risk of out-of-memory situations, but also improves memory-intensive workloads.
2: The Linux kernel will not overcommit memory, and only allocate as much memory as defined in overcommit_ratio.

.

Please try to change this setting as a superuser:

# echo 0 > /proc/sys/vm/overcommit_memory

.

Also for more detailed analysis can you please attach the JVM crash file "/home/centos/hs_err_pid4314.log"

[#3]

Hi @Jay Kumar SenSharma

on the named node free-m gives

total used free shared buff/cache available

14878 14088 250 198 540 325

Swap: 0 0 0

On a data node:

total used free shared buff/cache available

14878 5085 872 24 8920 9434

Swap: 0 0 0

/proc/sys/vm/overcommit_memory was set to 0

1. # echo $JAVA_OPTIONS
2. # echo $HADOOP_OPTS
3. # echo $HADOOP_CLIENT_OPTS

were all undefined

the log shows:

--------------- S Y S T E M --------------- OS:CentOS Linux release 7.2.1511 (Core) uname:Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 libc:glibc 2.17 NPTL 2.17 rlimit: STACK 8192k, CORE 0k, NPROC 59369, NOFILE 4096, AS infinity load average:2.56 1.96 2.06 /proc/meminfo: MemTotal: 15235812 kB MemFree: 157172 kB MemAvailable: 109724 kB Buffers: 0 kB Cached: 219468 kB SwapCached: 0 kB Active: 14636256 kB Inactive: 76244 kB Active(anon): 14626136 kB Inactive(anon): 67536 kB Active(file): 10120 kB Inactive(file): 8708 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 412 kB Writeback: 0 kB AnonPages: 14496064 kB Mapped: 40920 kB Shmem: 198124 kB Slab: 130892 kB SReclaimable: 55204 kB SUnreclaim: 75688 kB KernelStack: 55232 kB PageTables: 57976 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 7617904 kB Committed_AS: 27413956 kB VmallocTotal: 34359738367 kB VmallocUsed: 167920 kB VmallocChunk: 34359566840 kB HardwareCorrupted: 0 kB AnonHugePages: 8243200 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 94196 kB DirectMap2M: 3051520 kB DirectMap1G: 14680064 kB

[#4]

@Jay Kumar SenSharma....I just realized you had asked for the log file for this. Apologies for missing that. I have attached.

hs-err-pid4314log.txt

[#5]

This issue was likely caused by installing all master components (Named Node, RM, Hive Server 2, Druid, etc) on a single node. Redeploying a cluster with the master components distributed across 3 nodes resulted in a more stable cluster -- no more out of memory errors.