Support Questions

Find answers, ask questions, and share your expertise

HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Expert Contributor

Hello

HDP 2.3.2 & Ambari 2.1.2.1 that was upgraded to Ambari 2.4.2.0

After upgrading Ambari to 2.4.2.0 when i start Hive service the operation output is red with an error - but Hive is green, up & running without any problems.

Here is the error:

10264-snap-2016-12-13-at-113045.png

But this is the actual result - the restart indicator still exists and Hive is green. I'm also able to connect to hive using beeline and work as usual.

10265-snap-2016-12-13-at-113102.png

Service Check produces the following result

10266-snap-2016-12-13-at-162256.png

The hiveserver2.log is free of errors and Hive is working properly ! I can live with this but i'm afraid that i will have problems with upgrading the HDP because one of the prerequisites is successful service check...

Any ideas ?

1 ACCEPTED SOLUTION

@Adi Jabkowsky

Please run the below command:

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 

.

And place the following symlink on all hosts (specially hive host)

 # ls -l /usr/share/java/mysql-connector-java.jar
    lrwxrwxrwx 1 root root 31 Dec  2 07:48 /usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar

.

Looks like ambari is removing the mysql-connector jars symlink.

The changes that i suggested above to create the symlink is because it is not the "hiveserver2" failing but the "hive_service.py" is failing. Which is looking for the mysql jar.

https://github.com/apache/ambari/blob/release-2.4.2/ambari-server/src/main/resources/common-services...

View solution in original post

14 REPLIES 14

@Adi Jabkowsky

From your 3rd error screen shot it looks like the "zookeeper.jar" path is set incorrectly.

Please correct the "templeton.libjars" Property value as following:

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- In the image we see that the zookeeper.jar path is incorrect.

- The "templeton.libjars" can be set via ambari

"Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site"

Expert Contributor

Hi @jss

Thank you for your reply

The "templeton.libjars" Property hasen't been changed so i'm kind of baffled why i need to "correct" it. I do agree that the path which the service check is looking seems corrupt:

/usr/hdp/2.3.2.0-2950/hive/lib/hive-common.jar/zookeeper.jar

hive-common.jar/zookeeper.jar (jar file \ jar file)

It seems like a bug in the service check.... there can't be a .jar file under a .jar file... Or am i missing here something ?

@Adi Jabkowsky

You encountered a bug regarding the "templeton.libjars" Property. Please file it. Also please correct that "templeton.libjars" Property value and the service check issue should be fixed.

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- Else please compare the value of "templeton.libjars" property from any of your working environment and you will find that it will be same as above.

Expert Contributor
@jss

I've compared to other working environments and the value is the same in all of them:

/usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar

@Adi Jabkowsky

It seems Hive Metastore is having an issue with hive DB.

might be with following reason:

The HIVE metastore is a different host to the host running HiveServer2. Permissions on the metastore do not allow connections from the HiveServer2 host.

Expert Contributor
@Divakar Annapureddy

Thank you for replying!

The HIVE metastore is on the same host as the running HiveServer2. Nothing was changed in the hive mysql \ metastore- only Ambari was upgraded. That's the only change. Also - if there was a problem with connection between the HS2 to the metastore - Hive wouldn't work, but fortunately it works without a problem!

I have NO problems with restarting Hive nor working with it. I can connect to any Hive DBs, query as usual. Permissions are working. Everything seems normal and green. My problem is that the restart indicator doesn't go away + restarting produces error although it succeeds with restarting + service check fails.

just made small modification in any of your config file and give a try to re-start that service

Expert Contributor

in your error logs , it says that the com.mysql.jdbc.driver could not be located. can you see if the mysql jdbc jar is in the location used by the service check. As per logs it should be at /usr/hdp/current/hive-server2/lib/mysql-connector-java.jar.

@Adi Jabkowsky

So in general there seems to be two issues (both need to be addressed)

1. Incorrect "templeton.libjars"

>> Please correct the "templeton.libjars" Property value as following:

   /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

2. mysql connection JDBC jar is missing >> pelase check if the JAR is present or nor? Or If missing then you can copy it from other installation/host.

.

Expert Contributor

@jss

1. Corrected the path to /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

Still same error in service check. Seems like a straight up bug cause it's searching for hive-common.jar/zookeeper.jar.

2. mysql-connector-java.jar was missing from /usr/hdp/current/hive-server2/lib

I found one in /var/lib/ambari-server/resources/mysql-connector-java.jar and copied it to /usr/hdp/current/hive-server2/lib

After restarting the Hive service - it was gone. I checked it twice.

Here is the before & after using same command " ll | grep mysql"

10267-snap-2016-12-13-at-173029.png

How can Hive work properly if it has problems connecting to the metastore ?

@Adi Jabkowsky

Good to know that the mysql related error is gone.

However regarding the "templeton.libjars" jar issue "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar" i have seen some environments where after upgrade when we fix the "templeton.libjars" path then the issue gets resolved.

- Can you please double check if you have mode the correct changed to it inside the "Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site" config?

- After that you will need to restart the services that requires restart (including the "webhcat").

- Also can you please share the latest screenshot?

Expert Contributor

@jss

I probably didn't explain myself correct:

The mysql error isn't gone - the mysql-connector-java.jar file is gone ! I copied it to the /usr/hdp/current/hive-server2/lib path but after restarting the Hive service - the file is gone! It is as if it is being deleted by the restart itself . Look at the screenshot of before and after (The shell in which i did "ll | grep mysql") - it's there, and after restarting - it's not there anymore. The file is gone. Vanished. Disappeared. Regarding the templeton.libjars here is the screenshot after the fix:

10281-snap-2016-12-13-at-175807.png

@Adi Jabkowsky

Please run the below command:

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar 

.

And place the following symlink on all hosts (specially hive host)

 # ls -l /usr/share/java/mysql-connector-java.jar
    lrwxrwxrwx 1 root root 31 Dec  2 07:48 /usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar

.

Looks like ambari is removing the mysql-connector jars symlink.

The changes that i suggested above to create the symlink is because it is not the "hiveserver2" failing but the "hive_service.py" is failing. Which is looking for the mysql jar.

https://github.com/apache/ambari/blob/release-2.4.2/ambari-server/src/main/resources/common-services...

Expert Contributor

After running ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Everything works - restart & service check !

Thank you so much !

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.