HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

[#1]

Hello

HDP 2.3.2 & Ambari 2.1.2.1 that was upgraded to Ambari 2.4.2.0

After upgrading Ambari to 2.4.2.0 when i start Hive service the operation output is red with an error - but Hive is green, up & running without any problems.

Here is the error:

But this is the actual result - the restart indicator still exists and Hive is green. I'm also able to connect to hive using beeline and work as usual.

Service Check produces the following result

The hiveserver2.log is free of errors and Hive is working properly ! I can live with this but i'm afraid that i will have problems with upgrading the HDP because one of the prerequisites is successful service check...

Any ideas ?

[#2]

@jss

1. Corrected the path to /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

Still same error in service check. Seems like a straight up bug cause it's searching for hive-common.jar/zookeeper.jar.

2. mysql-connector-java.jar was missing from /usr/hdp/current/hive-server2/lib

I found one in /var/lib/ambari-server/resources/mysql-connector-java.jar and copied it to /usr/hdp/current/hive-server2/lib

After restarting the Hive service - it was gone. I checked it twice.

Here is the before & after using same command " ll | grep mysql"

How can Hive work properly if it has problems connecting to the metastore ?

[#3]

@Adi Jabkowsky

Good to know that the mysql related error is gone.

However regarding the "templeton.libjars" jar issue "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar" i have seen some environments where after upgrade when we fix the "templeton.libjars" path then the issue gets resolved.

- Can you please double check if you have mode the correct changed to it inside the "Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site" config?

- After that you will need to restart the services that requires restart (including the "webhcat").

- Also can you please share the latest screenshot?

[#4]

@jss

I probably didn't explain myself correct:

The mysql error isn't gone - the mysql-connector-java.jar file is gone ! I copied it to the /usr/hdp/current/hive-server2/lib path but after restarting the Hive service - the file is gone! It is as if it is being deleted by the restart itself . Look at the screenshot of before and after (The shell in which i did "ll | grep mysql") - it's there, and after restarting - it's not there anymore. The file is gone. Vanished. Disappeared. Regarding the templeton.libjars here is the screenshot after the fix:

[#5 · Accepted solution]

@Adi Jabkowsky

Please run the below command:

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 

.

And place the following symlink on all hosts (specially hive host)

 # ls -l /usr/share/java/mysql-connector-java.jar
    lrwxrwxrwx 1 root root 31 Dec  2 07:48 /usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar

.

Looks like ambari is removing the mysql-connector jars symlink.

The changes that i suggested above to create the symlink is because it is not the "hiveserver2" failing but the "hive_service.py" is failing. Which is looking for the mysql jar.

https://github.com/apache/ambari/blob/release-2.4.2/ambari-server/src/main/resources/common-services...

[#6]

After running ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Everything works - restart & service check !

Thank you so much !