Support Questions
Find answers, ask questions, and share your expertise

HiveServer2 Increase heap size

HiveServer2 Increase heap size

Expert Contributor

I increased heap size of HS2 to 12 GB in hive-env using ambari console. But when I grep the process_id in command line, I can different Xmx configuration. I am not sure where other Xmx configuration values are picked up .

Here is the process_id information.

	hive     10702     1  0 01:24 ?        00:09:28 /usr/java/default/bin/java -Xmx1024m -Dhdp.version=2.2.6.0-2800 -Djava.net.preferIPv4Stack=true -XX:NewRatio=12 -Xms10m -Xmx12288m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dhdp.version=2.2.6.0-2800 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.2.6.0-2800/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.2.6.0-2800/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.2.6.0-2800/hive/lib/hive-service-0.14.0.2.2.6.0-2800.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive

This is how I added the config.

 if [ "$SERVICE" = "hiveserver2" ]; then
   if [ -z "$DEBUG" ]; then
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx12288m -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
   else
     export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xmx12288m  -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:-UseGCOverheadLimit"
   fi
 fi

	

Which parameter will be applied at runtime ? Is it 1 GB or 12 GB ? How do I fix or configure 12 GB to hive server ?

6 REPLIES 6

Re: HiveServer2 Increase heap size

@Anandha L Ranganathan I see your heapsize values there as you have configured.

/usr/java/default/bin/java -Xmx1024m-Dhdp.version=2.2.6.0-2800-Djava.net.preferIPv4Stack=true-XX:NewRatio=12-Xms10m-Xmx12288m-XX:MaxHeapFreeRatio=40-XX:MinHeapFreeRatio=15-XX:+UseParNewGC-XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/tmp -Dhdp.version=2.2.6.0-2800 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.2.6.0-2800/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.2.6.0-2800/hadoop/lib/native-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true-Xmx1024m-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar/usr/hdp/2.2.6.0-2800/hive/lib/hive-service-0.14.0.2.2.6.0-2800.jar org.apache.hive.service.server.HiveServer2--hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive

Re: HiveServer2 Increase heap size

Expert Contributor

But it also has other Xmx parameters ? Will one of them will take precedence ?

/usr/java/default/bin/java -Xmx1024m-Dhdp.version=2.2.6.0-2800-Djava.net.preferIPv4Stack=true-XX:NewRatio=12-Xms10m-Xmx12288m-XX:MaxHeapFreeRatio=40-XX:MinHeapFreeRatio=15-XX:+UseParNewGC-XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/tmp -Dhdp.version=2.2.6.0-2800 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.2.6.0-2800/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.2.6.0-2800/hadoop/lib/native-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true-Xmx1024m-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar/usr/hdp/2.2.6.0-2800/hive/lib/hive-service-0.14.0.2.2.6.0-2800.jar org.apache.hive.service.server.HiveServer2--hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris= -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive

Re: HiveServer2 Increase heap size

Contributor

First -Xmx1G is for client sessions, Second -Xmx12G is for hiveserver2.

Re: HiveServer2 Increase heap size

Expert Contributor

Still confusing , from where the Xmx=1024 values are getting inherited when we specify We already twice had OOM exception and we restarted the hiveserver.

Here is the jmap heap size of hiveserver2 process.

Attaching to process ID 18505, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.71-b15


using parallel threads in the new generation.
using thread-local object allocation.
Mark Sweep Compact GC


Heap Configuration:
   MinHeapFreeRatio         = 15
   MaxHeapFreeRatio         = 40
   MaxHeapSize              = 1073741824 (1024.0MB)
   NewSize                  = 1310720 (1.25MB)
   MaxNewSize               = 82575360 (78.75MB)
   OldSize                  = 9175040 (8.75MB)
   NewRatio                 = 12
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)


Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 1179648 (1.125MB)
   used     = 431768 (0.41176605224609375MB)
   free     = 747880 (0.7132339477539062MB)
   36.60142686631944% used
Eden Space:
   capacity = 1048576 (1.0MB)
   used     = 300944 (0.2870025634765625MB)
   free     = 747632 (0.7129974365234375MB)
   28.70025634765625% used
From Space:
   capacity = 131072 (0.125MB)
   used     = 130824 (0.12476348876953125MB)
   free     = 248 (2.3651123046875E-4MB)
   99.810791015625% used
To Space:
   capacity = 131072 (0.125MB)
   used     = 0 (0.0MB)
   free     = 131072 (0.125MB)
   0.0% used
tenured generation:
   capacity = 15015936 (14.3203125MB)
   used     = 12850608 (12.255294799804688MB)
   free     = 2165328 (2.0650177001953125MB)
   85.57980002045826% used

Here is log message provided by ambari-agent to start the hive-server.

2016-05-06 00:56:49,341 - Execute['/var/lib/ambari-agent/data/tmp/start_hiveserver2_script /var/log/hive/hive-server2.out /var/log/hive/hive-server2.log /var/run/hive/hive-server.pid /etc/hive/conf.server /var/log/hive'] {'environment': {'HIVE_BIN': 'hive', 'JAVA_HOME': '/usr/java/default', 'HADOOP_HOME': '/usr/hdp/current/hadoop-client'}, 'not_if': "ls /var/run/hive/hive-server.pid >/dev/null 2>&1 && ps -p `ambari-sudo.sh su hive -l -s /bin/bash -c 'cat /var/run/hive/hive-server.pid'` >/dev/null 2>&1", 'user': 'hive', 'path': ['/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hive-server2/bin:/usr/hdp/current/hadoop-client/bin']}

	

Re: HiveServer2 Increase heap size

New Contributor

It seems that JVM treats the rightmost instance of an argument as the winner, so in your example rightmost value of Xmx is 1024m.

Xmx1024m-Dhdp.version=2.2.6.0-2800-Djava.net.preferIPv4Stack=true-XX:NewRatio=12-Xms10m-Xmx12288m-XX:MaxHeapFreeRatio=40-XX:MinHeapFreeRatio=15-XX:+UseParNewGC-XX:-UseGCOverheadLimit -XX:+HeapDumpOnOutOfMemoryError-XX:HeapDumpPath=/tmp -Dhdp.version=2.2.6.0-2800 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.2.6.0-2800/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.2.6.0-2800/hadoop/lib/native-Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true-Xmx1024m-

Value 1024m is taken from Hadoop maximum Java heap size (hadoop_heapsize) from Ambari (HDFS->Config->General), so it is possible to increase this parametr but other services also use it.

Instead of HADOOP_OPTS I've used HADOOP_CLIENT_OPTS and now new Xmx value is rightmost.

Unfortunately I couldn't find "official" instruction how to change Hiveserver heap size in Ambari.

Re: HiveServer2 Increase heap size

Guru

Once the values are entered in Ambari, they take effect only once the configurations are saved and restarted, as the new values are loaded. You might also consider looking at the values using "jinfo <pid> | egrep -i heap" to see the allocated max heap size. Changes made via Ambari are indeed reflected in hive-env.sh.

Changes made to hive-env.sh should also be made via Ambari.