question Random Alerts for Hive in Ambari in Archives of Support Questions (Read Only)

Random Alerts for Hive in Ambari

darpan_be — Wed, 23 Dec 2015 18:54:06 GMT

Hi,

After restarting the cluster randomly we see couple of red alerts in Ambari. Earlier also I remember seeing them sometime back and now again I see them again.

Can you suggest what could be going wrong?

I checked the ports 10001 and 9083 are open/in use.

HiveHive Metastore Process

====================================

Connection failed on host HIVE_HOST:10001 (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'"'"' ; ! beeline -u '"'"'jdbc:hive2://HIVE_HOST:10001/;transportMode=http;httpPath=cliservice;principal=hive/_HOST@REALM.COM'"'"' -e '"'"''"'"' 2>&1| awk '"'"'{print}'"'"'|grep -i -e '"'"'Connection refused'"'"' -e '"'"'Invalid URL'"'"''' was killed due timeout after 30 seconds)

HiveHiveServer2 Process

=============================

Metastore on HIVE_HOST failed (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/hdp/current/hive-metastore/bin'"'"' ; export HIVE_CONF_DIR='"'"'/usr/hdp/current/hive-metastore/conf/conf.server'"'"' ; hive --hiveconf hive.metastore.uris=thrift://HIVE_HOST:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 30 seconds)

Re: Random Alerts for Hive in Ambari

nsabharwal — Wed, 23 Dec 2015 19:19:59 GMT

@Darpan Patel I have seen this behavior during the network hicups or when cluster is overloaded.

Re: Random Alerts for Hive in Ambari

darpan_be — Wed, 23 Dec 2015 20:02:47 GMT

Well but network is stable here. And no jobs are running on the cluster!

Re: Random Alerts for Hive in Ambari

nsabharwal — Wed, 23 Dec 2015 20:07:37 GMT

@Darpan Patel I recommend opening a support case and let support take a look.

Re: Random Alerts for Hive in Ambari

darpan_be — Wed, 23 Dec 2015 20:19:11 GMT

Alright Neeraj. @Neeraj Sabharwal

Thank you again 🙂

Re: Random Alerts for Hive in Ambari

Jennyb — Fri, 12 Feb 2016 23:51:02 GMT

Hi did you ever get an answer to the time out errors? Is there a way to extend the timeout to allow them to finish?

Re: Random Alerts for Hive in Ambari

elkan1788 — Fri, 23 Sep 2016 16:01:23 GMT

Also I had meet the second problem? How can I update the connect timeout setting? There were test environment not enough resources so I don't mind it.