Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive OutOfMemoryError: unable to create new native thread

avatar
Contributor

Seeing below exception on running Hive TPCDS data gen (https://github.com/hortonworks/hive-testbench) for a scale of ~500G.

 

 

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread

 

 

 

Attached log for complete stacktrace.

 

Cluster Configuration :

16 Nodes / 12 Nodemanagers / 12 Datanodes

 

Per Node Config :

Cores : 40

Memory : 392GB

 

Ambari Configs changed from initial configs to improve performance :

Decided to set 10G as container size to utilise maximum cores per node (320G/10G = 32 containers using 1 Core/node. Hence ~32 Cores/node utilised)

YARN

yarn.nodemanager.resource.memory-mb = 329216 MB
yarn.scheduler.minimum-allocation-mb = 10240 MB
yarn.scheduler.maximum-allocation-mb = 329216 MB
MapReduce (All Heap Sizes : -Xmx8192m : 80% of container)
mapreduce.map.memory.mb = 10240 MB
mapreduce.reduce.memory.mb = 10240 MB
mapreduce.task.io.sort.mb = 1792 MB
yarn.app.mapreduce.am.resource.mb = 10240 MB

Hive

hive.tez.container.size = 10240MB

hive.auto.convert.join.noconditionaltask.size = 2027316838 B

hive.exec.reducers.bytes.per.reducer = 1073217536 B

Tez

tez.am.resource.memory.mb = 10240 MB

tez.am.resource.java.opts = -server -Xmx8192m

tez.task.resource.memory.mb = 10240 MB

tez.runtime.io.sort.mb = 2047 MB (~20% of container)
tez.runtime.unordered.output.buffer.size-mb = 768 MB (~10% of container)
tez.grouping.max-size = 2073741824 B
tez.grouping.min-size = 167772160 B
 
Any help would be greatly appreciated.
Referred https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step... for some tuning values.
3 REPLIES 3

avatar
Expert Contributor

Hi @ssulav ,

 

I see the below error in the HS2 logs shared:

 

ERROR [HiveServer2-Background-Pool: Thread-886]: SessionState (:()) - Vertex failed, vertexName=Map 1, vertexId=vertex_1571760131080_0019_1_00, diagnostics=[Task failed, taskId=task_1571760131080_0019_1_00_000380, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: unable to create new native thread

The above error is reported by the Yarn application.

This error is not related to any Hadoop or Yarn or Hive configuration. Rather this is an error received by the OS for now able to create new threads for the process.

You need to check the ulimits of the Yarn user on the NodeManager and ResourceManager nodes.

Though, it is more likely this error is coming from the NodeManager running the above mentioned Task ID.

You can identify the host where the above Task is running by searching for the vertex or Task ID in the Yarn application logs.

 

The job would be running as Yarn user and check for similar errors in the nodemanager logs for the same host.

You can try increase the ulimit for "ulimit -u" option.

avatar
Contributor

Thanks for reply @rohitmalhotra 
User Limit for yarn is set to 65536.
Is there any recommended highest value or shall I just make it unlimited? (It can have consequences?)

 

Edit : I tried setting unlimited. Still seeing same error.

avatar
Expert Contributor

You can try changing the limits from Ambari as well.

 

Under Ambari > Yarn Configs > Advanced:

 

Screenshot 2019-10-24 at 12.13.56 PM.png

 

Restart Yarn after increasing the limit.