question Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2 in Archives of Support Questions (Read Only)

HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Sun, 18 Aug 2019 12:48:51 GMT

Hello

HDP 2.3.2 & Ambari 2.1.2.1 that was upgraded to Ambari 2.4.2.0

After upgrading Ambari to 2.4.2.0 when i start Hive service the operation output is red with an error - but Hive is green, up & running without any problems.

Here is the error:

But this is the actual result - the restart indicator still exists and Hive is green. I'm also able to connect to hive using beeline and work as usual.

Service Check produces the following result

The hiveserver2.log is free of errors and Hive is working properly ! I can live with this but i'm afraid that i will have problems with upgrading the HDP because one of the prerequisites is successful service check...

Any ideas ?

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Former Member — Tue, 13 Dec 2016 22:33:32 GMT

@Adi Jabkowsky

From your 3rd error screen shot it looks like the "zookeeper.jar" path is set incorrectly.

Please correct the "templeton.libjars" Property value as following:

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- In the image we see that the zookeeper.jar path is incorrect.

- The "templeton.libjars" can be set via ambari

"Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site"

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

divakarreddy_a — Tue, 13 Dec 2016 22:44:05 GMT

@Adi Jabkowsky

It seems Hive Metastore is having an issue with hive DB.

might be with following reason:

The HIVE metastore is a different host to the host running HiveServer2. Permissions on the metastore do not allow connections from the HiveServer2 host.

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Tue, 13 Dec 2016 22:45:59 GMT

Hi @jss

Thank you for your reply

The "templeton.libjars" Property hasen't been changed so i'm kind of baffled why i need to "correct" it. I do agree that the path which the service check is looking seems corrupt:

/usr/hdp/2.3.2.0-2950/hive/lib/hive-common.jar/zookeeper.jar

hive-common.jar/zookeeper.jar (jar file \ jar file)

It seems like a bug in the service check.... there can't be a .jar file under a .jar file... Or am i missing here something ?

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Former Member — Tue, 13 Dec 2016 22:50:05 GMT

@Adi Jabkowsky

You encountered a bug regarding the "templeton.libjars" Property. Please file it. Also please correct that "templeton.libjars" Property value and the service check issue should be fixed.

/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

- Else please compare the value of "templeton.libjars" property from any of your working environment and you will find that it will be same as above.

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Tue, 13 Dec 2016 22:50:56 GMT

@Divakar Annapureddy

Thank you for replying!

The HIVE metastore is on the same host as the running HiveServer2. Nothing was changed in the hive mysql \ metastore- only Ambari was upgraded. That's the only change. Also - if there was a problem with connection between the HS2 to the metastore - Hive wouldn't work, but fortunately it works without a problem!

I have NO problems with restarting Hive nor working with it. I can connect to any Hive DBs, query as usual. Permissions are working. Everything seems normal and green. My problem is that the restart indicator doesn't go away + restarting produces error although it succeeds with restarting + service check fails.

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Tue, 13 Dec 2016 22:54:47 GMT

@jss

I've compared to other working environments and the value is the same in all of them:

/usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

knarayanan — Tue, 13 Dec 2016 22:56:23 GMT

in your error logs , it says that the com.mysql.jdbc.driver could not be located. can you see if the mysql jdbc jar is in the location used by the service check. As per logs it should be at /usr/hdp/current/hive-server2/lib/mysql-connector-java.jar.

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Former Member — Tue, 13 Dec 2016 23:03:23 GMT

@Adi Jabkowsky

So in general there seems to be two issues (both need to be addressed)

1. Incorrect "templeton.libjars"

>> Please correct the "templeton.libjars" Property value as following:

   /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

2. mysql connection JDBC jar is missing >> pelase check if the JAR is present or nor? Or If missing then you can copy it from other installation/host.

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

divakarreddy_a — Tue, 13 Dec 2016 23:06:19 GMT

just made small modification in any of your config file and give a try to re-start that service

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Sun, 18 Aug 2019 12:48:32 GMT

@jss

1. Corrected the path to /usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar

Still same error in service check. Seems like a straight up bug cause it's searching for hive-common.jar/zookeeper.jar.

2. mysql-connector-java.jar was missing from /usr/hdp/current/hive-server2/lib

I found one in /var/lib/ambari-server/resources/mysql-connector-java.jar and copied it to /usr/hdp/current/hive-server2/lib

After restarting the Hive service - it was gone. I checked it twice.

Here is the before & after using same command " ll | grep mysql"

How can Hive work properly if it has problems connecting to the metastore ?

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Former Member — Tue, 13 Dec 2016 23:53:02 GMT

@Adi Jabkowsky

Good to know that the mysql related error is gone.

However regarding the "templeton.libjars" jar issue "/usr/hdp/${hdp.version}/zookeeper/zookeeper.jar,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar" i have seen some environments where after upgrade when we fix the "templeton.libjars" path then the issue gets resolved.

- Can you please double check if you have mode the correct changed to it inside the "Hive" ==> "config" ==> "Advanced" ==> "Advanced webhcat-site" config?

- After that you will need to restart the services that requires restart (including the "webhcat").

- Also can you please share the latest screenshot?

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Sun, 18 Aug 2019 12:48:24 GMT

@jss

I probably didn't explain myself correct:

The mysql error isn't gone - the mysql-connector-java.jar file is gone ! I copied it to the /usr/hdp/current/hive-server2/lib path but after restarting the Hive service - the file is gone! It is as if it is being deleted by the restart itself . Look at the screenshot of before and after (The shell in which i did "ll | grep mysql") - it's there, and after restarting - it's not there anymore. The file is gone. Vanished. Disappeared. Regarding the templeton.libjars here is the screenshot after the fix:

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Former Member — Wed, 14 Dec 2016 00:26:22 GMT

@Adi Jabkowsky

Please run the below command:

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

And place the following symlink on all hosts (specially hive host)

 # ls -l /usr/share/java/mysql-connector-java.jar
    lrwxrwxrwx 1 root root 31 Dec  2 07:48 /usr/share/java/mysql-connector-java.jar -> mysql-connector-java-5.1.17.jar

Looks like ambari is removing the mysql-connector jars symlink.

The changes that i suggested above to create the symlink is because it is not the "hiveserver2" failing but the "hive_service.py" is failing. Which is looking for the mysql jar.

https://github.com/apache/ambari/blob/release-2.4.2/ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py#L151-L155

Re: HiveServer2 shows failure - but starts after upgrading to Ambari 2.4.2

Adija1 — Wed, 14 Dec 2016 00:44:19 GMT

After running ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

Everything works - restart & service check !

Thank you so much !