question Re: hive + many hive process was open on namenode machine and take a lot memory in Support Questions

hive + many hive process was open on namenode machine and take a lot memory

mike_bronson7 — Mon, 09 Jul 2018 03:09:22 GMT

we have ambari cluster version 2.6.0.x

we noitce that free memory on master01 machine is 0 ,

and that because many of the following process ( from ps -ef | grep java )

ambari-+ 65369 65322  0 Jul06 ?        00:01:52 /usr/jdk64/jdk1.8.0_112/bin/java -Xmx1024m -Dhdp.version=2.6.0.3-8 -Djava.net.preferIPv4Stack=true -Dhdvar/log/hadoop/ambari-qa -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.6.0.3-8/hadoop -Dhadoop.id.str=ambari-qa -Dhadoop.root.logger=INFO,c.6.0.3-8/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.6.0.3-8/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -.config.file=/usr/hdp/2.6.0.3-8/hive/conf/parquet-logging.properties -Dlog4j.configuration=beeline-log4j.properties -Dhadoop.security.logger=INFO,NullA/usr/hdp/2.6.0.3-8/hive/lib/hive-beeline-1.2.1000.2.6.0.3-8.jar org.apache.hive.beeline.BeeLine -u jdbc:hive2://master01.sys748.com:10000/;transportMo

any idea why all this process ( around 350 process was opened ? ) , and take all memory on master01 machine

[root@master01 ~]# ps -ef | grep java | wc -l
359


other way to show the process :

 ps -ef |   sed 's/-D[^ ]*//g;s/-X[^ ]*//g;s#^.*/bin/java##g;s/[^ ]*.jar//g;s/^[ ]*//g' |more



ambari-+ 50648 50646  0 Jul06 ?        00:00:00 -bash -c export  PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/
*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin
/:/usr/sbin/' ; ! beeline -u 'jdbc:hive2://master01.sys748.com:10000/;transportMode=binary'  -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused'
 -e 'Invalid URL'

Re: hive + many hive process was open on namenode machine and take a lot memory

vmurakami — Mon, 09 Jul 2018 11:51:07 GMT

Hi @Michael Bronson!
Could you share with us the output of these commands?

jstack -l <PID_HS2>
pstree -p hive

Hope this helps!

Re: hive + many hive process was open on namenode machine and take a lot memory

mike_bronson7 — Mon, 09 Jul 2018 13:18:23 GMT

not have the jstack command on my linux machine , should I download this cli from redhat repo?

Re: hive + many hive process was open on namenode machine and take a lot memory

jsensharma — Mon, 09 Jul 2018 14:00:27 GMT

@Michael Bronson

If you see that there are many "beeline" processes are being created by ambari or ambari-qa user then it might be the Hive Alert checker scripts which might be leaving opened Beeline connections ... which might be causing the issue.

Please see the script [1]

On Ambari Server host

/var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py

On Agent hosts

/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py

May be for some time you can try disabling the "HiveServer 2 Process" Alert from Ambari UI and then kill those beeling process manually using command line to see if it fixes the issue.

Ideally there were some old issues reported like that but those were already supposed to be fixed ambari 2.5.0 onwards like: https://issues.apache.org/jira/browse/AMBARI-18286

[1] https://github.com/apache/ambari/blob/release-2.6.1/ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py#L275

Re: hive + many hive process was open on namenode machine and take a lot memory

jsensharma — Mon, 09 Jul 2018 14:04:04 GMT

@Michael Bronson

Additionally please check if there is any issue in connecting to HiveSerevr2 using Beeline manually? This will give us some idea like how much time does the beeline connection takes. If it takes more than default hardcoded 30 secodns then please try to increase the timeout to a higher value like 60 in the same script:

# grep 'timeout=30' /var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py

# grep 'timeout=30' /var/lib/ambari-server/resources/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_thrift_port.py