Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Hive OutOfMemoryError: unable to create new native thread

Cloudera Employee

Seeing below exception on running Hive TPCDS data gen ( for a scale of ~500G.



Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.OutOfMemoryError: unable to create new native thread




Attached log for complete stacktrace.


Cluster Configuration :

16 Nodes / 12 Nodemanagers / 12 Datanodes


Per Node Config :

Cores : 40

Memory : 392GB


Ambari Configs changed from initial configs to improve performance :

Decided to set 10G as container size to utilise maximum cores per node (320G/10G = 32 containers using 1 Core/node. Hence ~32 Cores/node utilised)


yarn.nodemanager.resource.memory-mb = 329216 MB
yarn.scheduler.minimum-allocation-mb = 10240 MB
yarn.scheduler.maximum-allocation-mb = 329216 MB
MapReduce (All Heap Sizes : -Xmx8192m : 80% of container) = 10240 MB
mapreduce.reduce.memory.mb = 10240 MB = 1792 MB = 10240 MB


hive.tez.container.size = 10240MB = 2027316838 B

hive.exec.reducers.bytes.per.reducer = 1073217536 B

Tez = 10240 MB = -server -Xmx8192m

tez.task.resource.memory.mb = 10240 MB = 2047 MB (~20% of container)
tez.runtime.unordered.output.buffer.size-mb = 768 MB (~10% of container)
tez.grouping.max-size = 2073741824 B
tez.grouping.min-size = 167772160 B
Any help would be greatly appreciated.
Referred for some tuning values.


Hi @ssulav ,


I see the below error in the HS2 logs shared:


ERROR [HiveServer2-Background-Pool: Thread-886]: SessionState (:()) - Vertex failed, vertexName=Map 1, vertexId=vertex_1571760131080_0019_1_00, diagnostics=[Task failed, taskId=task_1571760131080_0019_1_00_000380, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: unable to create new native thread

The above error is reported by the Yarn application.

This error is not related to any Hadoop or Yarn or Hive configuration. Rather this is an error received by the OS for now able to create new threads for the process.

You need to check the ulimits of the Yarn user on the NodeManager and ResourceManager nodes.

Though, it is more likely this error is coming from the NodeManager running the above mentioned Task ID.

You can identify the host where the above Task is running by searching for the vertex or Task ID in the Yarn application logs.


The job would be running as Yarn user and check for similar errors in the nodemanager logs for the same host.

You can try increase the ulimit for "ulimit -u" option.

Cloudera Employee

Thanks for reply @rohitmalhotra 
User Limit for yarn is set to 65536.
Is there any recommended highest value or shall I just make it unlimited? (It can have consequences?)


Edit : I tried setting unlimited. Still seeing same error.


You can try changing the limits from Ambari as well.


Under Ambari > Yarn Configs > Advanced:


Screenshot 2019-10-24 at 12.13.56 PM.png


Restart Yarn after increasing the limit.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.