HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

[#1]

Hello

HDP 2.3.2 & Ambari 2.1.2.1 that was upgraded to Ambari 2.4.2.0

After upgrading Ambari to 2.4.2.0 when i start Hive service the operation output is red with an error - but Hive is green, up & running without any problems.

Here is the error:

But this is the actual result - the restart indicator still exists and Hive is green. I'm also able to connect to hive using beeline and work as usual.

Service Check produces the following result

The hiveserver2.log is free of errors and Hive is working properly ! I can live with this but i'm afraid that i will have problems with upgrading the HDP because one of the prerequisites is successful service check...

Any ideas ?

[#2]

@Adi Jabkowsky

From your 3rd error screen shot it looks like the "zookeeper.jar" path is set incorrectly.

Please correct the "templeton.libjars" Property value as following:

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- In the image we see that the zookeeper.jar path is incorrect.

- The "templeton.libjars" can be set via ambari

"Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site"

[#3]

Hi @jss

Thank you for your reply

The "templeton.libjars" Property hasen't been changed so i'm kind of baffled why i need to "correct" it. I do agree that the path which the service check is looking seems corrupt:

/usr/hdp/2.3.2.0-2950/hive/lib/hive-common.jar/zookeeper.jar

hive-common.jar/zookeeper.jar (jar file \ jar file)

It seems like a bug in the service check.... there can't be a .jar file under a .jar file... Or am i missing here something ?

[#4]

@Adi Jabkowsky

You encountered a bug regarding the "templeton.libjars" Property. Please file it. Also please correct that "templeton.libjars" Property value and the service check issue should be fixed.

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- Else please compare the value of "templeton.libjars" property from any of your working environment and you will find that it will be same as above.

[#5]

@jss

I've compared to other working environments and the value is the same in all of them:

/usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar

[#6]

@Adi Jabkowsky

It seems Hive Metastore is having an issue with hive DB.

might be with following reason:

The HIVE metastore is a different host to the host running HiveServer2. Permissions on the metastore do not allow connections from the HiveServer2 host.

[#7]

@Divakar Annapureddy

Thank you for replying!

The HIVE metastore is on the same host as the running HiveServer2. Nothing was changed in the hive mysql \ metastore- only Ambari was upgraded. That's the only change. Also - if there was a problem with connection between the HS2 to the metastore - Hive wouldn't work, but fortunately it works without a problem!

I have NO problems with restarting Hive nor working with it. I can connect to any Hive DBs, query as usual. Permissions are working. Everything seems normal and green. My problem is that the restart indicator doesn't go away + restarting produces error although it succeeds with restarting + service check fails.

[#8]

just made small modification in any of your config file and give a try to re-start that service

[#9]

in your error logs , it says that the com.mysql.jdbc.driver could not be located. can you see if the mysql jdbc jar is in the location used by the service check. As per logs it should be at /usr/hdp/current/hive-server2/lib/mysql-connector-java.jar.

[#10]

@Adi Jabkowsky

So in general there seems to be two issues (both need to be addressed)

1. Incorrect "templeton.libjars"

>> Please correct the "templeton.libjars" Property value as following:

   /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

2. mysql connection JDBC jar is missing >> pelase check if the JAR is present or nor? Or If missing then you can copy it from other installation/host.

.