Datanode out of memory

[crazy-dadong]

Hello everyone!

My cluster have 4 datanode. Each Datanode has 32G RAM.

Recently, the Datanode has been reported Exception Out Of Memory.

I looed up some document, just mentioned namenode memory situations.

CDH Manager，Host memory usage chart display physical memory is full, but The role of resident memory less than 1G.

i don't know why it took so much of physical memory. 

On the system currently has 12 million blocks

I have turn off all other services,just run the hdfs service.

[Harsh J Cloudera Employee]

You've mentioned the RAM of the machine your DataNode is assigned to run
on, however what is your configured DataNode Java JVM heap size? You could
try raising it by 1 GB from its current value, to resolve this.

Also, what's the entire Out of Memory message, 'cause "unable to create a
new native thread" (or summat) is entirely different than "Java heap space"
in what it implies (nproc limit issue vs. actual heap memory exhaustion).