HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

[#1]

Hello

HDP 2.3.2 & Ambari 2.1.2.1 that was upgraded to Ambari 2.4.2.0

After upgrading Ambari to 2.4.2.0 when i start Hive service the operation output is red with an error - but Hive is green, up & running without any problems.

Here is the error:

10264-snap-2016-12-13-at-113045.png

But this is the actual result - the restart indicator still exists and Hive is green. I'm also able to connect to hive using beeline and work as usual.

10265-snap-2016-12-13-at-113102.png

Service Check produces the following result

10266-snap-2016-12-13-at-162256.png

The hiveserver2.log is free of errors and Hive is working properly ! I can live with this but i'm afraid that i will have problems with upgrading the HDP because one of the prerequisites is successful service check...

Any ideas ?

[#2]

@jss

1. Corrected the path to /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

Still same error in service check. Seems like a straight up bug cause it's searching for hive-common.jar/zookeeper.jar.

2. mysql-connector-java.jar was missing from /usr/hdp/current/hive-server2/lib

I found one in /var/lib/ambari-server/resources/mysql-connector-java.jar and copied it to /usr/hdp/current/hive-server2/lib

After restarting the Hive service - it was gone. I checked it twice.

Here is the before & after using same command " ll | grep mysql"

10267-snap-2016-12-13-at-173029.png

How can Hive work properly if it has problems connecting to the metastore ?

[#3]

@Adi Jabkowsky

Good to know that the mysql related error is gone.

However regarding the "templeton.libjars" jar issue "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar" i have seen some environments where after upgrade when we fix the "templeton.libjars" path then the issue gets resolved.

- Can you please double check if you have mode the correct changed to it inside the "Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site" config?

- After that you will need to restart the services that requires restart (including the "webhcat").

- Also can you please share the latest screenshot?

[#4]

@jss

I probably didn't explain myself correct:

The mysql error isn't gone - the mysql-connector-java.jar file is gone ! I copied it to the /usr/hdp/current/hive-server2/lib path but after restarting the Hive service - the file is gone! It is as if it is being deleted by the restart itself . Look at the screenshot of before and after (The shell in which i did "ll | grep mysql") - it's there, and after restarting - it's not there anymore. The file is gone. Vanished. Disappeared. Regarding the templeton.libjars here is the screenshot after the fix:

10281-snap-2016-12-13-at-175807.png

[#5 · Accepted solution]

@Adi Jabkowsky

Please run the below command:

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 

.

And place the following symlink on all hosts (specially hive host)

 # ls -l /usr/share/java/mysql-connector-java.jar
    lrwxrwxrwx 1 root root 31 Dec  2 07:48 /usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar

.

Looks like ambari is removing the mysql-connector jars symlink.

The changes that i suggested above to create the symlink is because it is not the "hiveserver2" failing but the "hive_service.py" is failing. Which is looking for the mysql jar.

https://github.com/apache/ambari/blob/release-2.4.2/ambari-server/src/main/resources/common-services...

[#6]

After running ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Everything works - restart & service check !

Thank you so much !