question Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running in Support Questions

Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

dineshe24 — Sun, 18 Aug 2019 11:20:54 GMT

Ambari dashboard is showing HiveServer2 and Hive Metastore is stopped, but from the command line I am able to login into hive cli and hiveserver2 using beeline client also.

I have 6 node cluster in which I have installed hive server in 2 nodes and hive client is installed in all 6 nodes. In node1 ambari dashboard it was showing hiveserver2 and hive metastore is stopped but in node2 it was started.

In node1:

In node2:

When checked the process id which was available in /var/run/hive/hive.pid and hive-server.pid, they are same.

[node1 ~]$ cd /var/run/hive
[node1.staging.iad hive]$ ls
hive.pid  hive-server.pid
[node1 hive]$ cat hive.pid
14037
[node1 hive]$ cat hive-server.pid
16654
[node1 hive]$ ps -ef| grep 14037
dinesh     564 32124  0 15:10 pts/0    00:00:00 grep 14037
hive     14037     1  0 Aug03 ?        00:01:45 /usr/lib/jvm/jre/bin/java -Xmx1024m -Dhdp.version=2.3.4.0-3485 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.0-3485 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.0-3485/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx1024m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.0-3485/hive/lib/hive-service-1.2.1.2.3.4.0-3485.jar org.apache.hadoop.hive.metastore.HiveMetaStore -hiveconf hive.log.file=hivemetastore.log -hiveconf hive.log.dir=/var/log/hive
[node1 hive]$ ps -ef| grep 16654
dinesh     621 32124  0 15:10 pts/0    00:00:00 grep 16654
hive     16654     1  0 Aug03 ?        00:01:32 /usr/lib/jvm/jre/bin/java -Xmx1024m -Dhdp.version=2.3.4.0-3485 -Djava.net.preferIPv4Stack=true -Dhdp.version=2.3.4.0-3485 -Dhadoop.log.dir=/var/log/hadoop/hive -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/hdp/2.3.4.0-3485/hadoop -Dhadoop.id.str=hive -Dhadoop.root.logger=INFO,console -Djava.library.path=:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.3.4.0-3485/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xmx1024m -XX:MaxPermSize=512m -Xmx512m -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar /usr/hdp/2.3.4.0-3485/hive/lib/hive-service-1.2.1.2.3.4.0-3485.jar org.apache.hive.service.server.HiveServer2 --hiveconf hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar -hiveconf hive.metastore.uris=  -hiveconf hive.log.file=hiveserver2.log -hiveconf hive.log.dir=/var/log/hive

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

KuldeepK — Thu, 04 Aug 2016 23:09:21 GMT

@Dinesh E

Can you restart Ambari agent and see if helps?

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

sbhat — Thu, 04 Aug 2016 23:27:46 GMT

@Dinesh E Here is an article on why this could happen. It explains why and steps to take when such a scenario occurs :

https://community.hortonworks.com/articles/18088/ambari-shows-hdp-services-to-be-down-whereas-they.html

Hope this helps!

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

dineshe24 — Fri, 05 Aug 2016 21:49:28 GMT

@Kuldeep Kulkarni

Thanks for your reply. Restarted ambari agent, but it doesn't worked out.

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

dineshe24 — Fri, 05 Aug 2016 21:58:56 GMT

@sbhat Thanks for you reply.

My issue falls under scenario 2 in that article.

But my hive.pid permission are correct i.e

-rw-r--r-- 1 hive hadoop 6 Aug  4 19:48 hive.pid
-rw-r--r-- 1 hive hadoop 6 Aug  4 19:48 hive-server.pid

And also my procees id's i.e hive-server.pid is matching with the process id when checked using

ps -ef|grep hive

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

jonas_friedrich — Tue, 16 Aug 2016 00:11:00 GMT

I have the same problems...Also the tipps didn´t work 😞

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

Shelton — Tue, 16 Aug 2016 04:03:42 GMT

@Dinesh E

It looks the following services are all in maintenance mode.

Hive Server2
Hive-Metastore
HCat client
Hive Client

Please Turn Off Maintenance Mode and let me know if that worked

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

jonas_friedrich — Tue, 16 Aug 2016 13:54:19 GMT

Didn´t work for me 😕

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

subhash_parise3 — Tue, 16 Aug 2016 14:04:40 GMT

Hi Dinesh,

I had same issue in my env for few months back. In my scenario hive service had started with other user (manually). not with ambari user. kill the procees id and again start the service using ambari console resolved my issue.

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

jonas_friedrich — Tue, 16 Aug 2016 16:56:49 GMT

Hi, @subhash parise. Thanks for your answer, but that didn´t worked for me ;-(

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

jayanta_das — Wed, 16 Nov 2016 01:09:57 GMT

@Dinesh, This is problem resolved after upgrading the HDP from 2.3.4 to 2.4.2

Re: Ambari shows HiveServer2 and Hive Metastore is stopped but they are running

dineshe24 — Wed, 16 Nov 2016 16:37:25 GMT

@Jayanta Das Thanks for the update.