question Re: hiveserver2 out of memory - what settings to use in Archives of Support Questions (Read Only)

hiveserver2 out of memory - what settings to use

maxjflanders — Sat, 04 Feb 2017 05:58:04 GMT

Our instance of hiveserver2 has been running out of memory about once per day requiring a restart. The machines running hive serve2 have 52gb of memory, but is there some kind of rule of thumb on how much memory to allocate, based on machine size??

ps ax | grep -i hiveserver2 returns:

hive 14668 1 0 Feb02 ? 00:02:16 /usr//bin/java -Xmx256m -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.7-4 -Dhdp.version=2.3.4.7-4 -Dhadoop.log.dir=/usr/hdp/2.3.4.7-4/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.7-4/hadoop -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/2.3.4.7-4/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.7-4/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.7-4/hive/lib/hive-service-1.2.1.2.3.4.7-4.jar org.apache.hive.service.server.HiveServer2

Re: hiveserver2 out of memory - what settings to use

mqureshi — Sat, 04 Feb 2017 06:37:23 GMT

@Maxwell Flanders

Rule of thumb varies on your specific use case. How many users are connecting concurrently? Regardless 256 MB is very low.

Bump it to at least 4GB and see if this resolves your issue. Change -Xmx256m to -Xmx4096m. I was able to find recommended values for HS2. Use this as a guideline:

Up to 20 concurrent connections	6 GB
Up to 10 concurrent connections	4 GB
Single connection	2 GB

Re: hiveserver2 out of memory - what settings to use

maxjflanders — Thu, 02 Mar 2017 01:20:42 GMT

That worked perfectly, thank you!