question Hive OutOfMemoryError: unable to create new native thread in Support Questions

question Hive OutOfMemoryError: unable to create new native thread in Support Questions https://community.cloudera.com/t5/Support-Questions/Hive-OutOfMemoryError-unable-to-create-new-native-thread/m-p/281034#M209113 <P>Seeing below exception on running Hive TPCDS data gen (<A href="https://github.com/hortonworks/hive-testbench" target="_blank" rel="noopener">https://github.com/hortonworks/hive-testbench</A>) for a scale of ~500G.</P> <P> </P> <P> </P> <LI-CODE lang="markup">Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread</LI-CODE> <P> </P> <P> </P> <P> </P> <P>Attached log for complete stacktrace.</P> <P> </P> <P><STRONG>Cluster Configuration :</STRONG></P> <P>16 Nodes / 12 Nodemanagers / 12 Datanodes</P> <P> </P> <P><STRONG>Per Node Config :</STRONG></P> <P>Cores : 40</P> <P>Memory : 392GB</P> <P> </P> <P><EM><STRONG><U>Ambari Configs changed from initial configs to improve performance :</U></STRONG></EM></P> <P>Decided to set 10G as container size to utilise maximum cores per node (320G/10G = 32 containers using 1 Core/node. Hence ~32 Cores/node utilised)</P> <P><STRONG><U>YARN</U></STRONG></P> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">yarn.nodemanager.resource.memory-mb = </SPAN></SPAN><SPAN class="compare-config-cell">329216 MB</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">yarn.scheduler.minimum-allocation-mb = </SPAN></SPAN><SPAN class="compare-config-cell">10240 MB</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">yarn.scheduler.maximum-allocation-mb = </SPAN></SPAN><SPAN class="compare-config-cell">329216 MB</SPAN></DIV> <DIV class="col-md-3 property-name-column"><U><STRONG><SPAN class="compare-config-cell">MapReduce (All Heap Sizes : <SPAN>-Xmx8192m : 80% of container)</SPAN></SPAN></STRONG></U></DIV> <DIV class="col-md-3 property-name-column"> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">mapreduce.map.memory.mb = </SPAN></SPAN><SPAN class="compare-config-cell">10240 MB</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">mapreduce.reduce.memory.mb = </SPAN></SPAN><SPAN>10240 MB</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">mapreduce.task.io.sort.mb = 1</SPAN></SPAN><SPAN class="compare-config-cell">792 MB</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">yarn.app.mapreduce.am.resource.mb = </SPAN></SPAN><SPAN>10240 MB</SPAN></DIV> </DIV> </DIV> </DIV> <P><U><STRONG>Hive</STRONG></U></P> <P>hive.tez.container.size = 10240MB</P> <P><SPAN>hive.auto.convert.join.noconditionaltask.size = 2027316838 B</SPAN></P> <P><SPAN>hive.exec.reducers.bytes.per.reducer = 1073217536 B</SPAN></P> <P><U><STRONG>Tez</STRONG></U></P> <P><SPAN>tez.</SPAN><SPAN>am.</SPAN><SPAN>resource.</SPAN><SPAN>memory.</SPAN><SPAN>mb = 10240 MB</SPAN></P> <P><SPAN>tez.am.resource.java.opts = -server -Xmx8192m</SPAN></P> <P><SPAN><SPAN class="compare-config-cell">tez.task.resource.memory.mb = 10240 MB</SPAN></SPAN></P> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">tez.runtime.io.sort.mb = </SPAN></SPAN><SPAN class="compare-config-cell">2047 MB (~20% of container)</SPAN></DIV> </DIV> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">tez.runtime.unordered.output.buffer.size-mb = </SPAN></SPAN><SPAN class="compare-config-cell">768 MB (~10% of container)</SPAN></DIV> <DIV class="col-md-3 property-name-column"> <DIV class="row comparison-row bottom-border has-compare-diffs"> <DIV class="col-md-3 property-name-column"><SPAN class="control-label"><SPAN class="ember-view">tez.grouping.max-size = </SPAN></SPAN><SPAN class="compare-config-cell">2073741824 B</SPAN></DIV> <DIV class="col-md-4 property-value-column"><SPAN>tez.</SPAN><SPAN>grouping.</SPAN><SPAN>min-size = </SPAN><SPAN>167772160 B</SPAN></DIV> <DIV class="col-md-4 property-value-column"> </DIV> <DIV class="col-md-4 property-value-column"><SPAN>Any help would be greatly appreciated.<BR />Referred <A href="https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step/ta-p/245279" target="_blank" rel="noopener">https://community.cloudera.com/t5/Community-Articles/Demystify-Apache-Tez-Memory-Tuning-Step-by-Step/ta-p/245279</A> for some tuning values.</SPAN></DIV> </DIV> </DIV> </DIV> Tue, 22 Oct 2019 20:49:38 GMT ssulav 2019-10-22T20:49:38Z