Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

(hive) How to dynamically set hive.aux.jars.path environment variable after HiveServer2 is started.

avatar

Getting the below question in the context of an application that creates tables with a custom storage handler:

When I tried setting jar locations on the fly using “set hive.aux.jars.path=<file urls>” on beeline prompt after connecting to HS2, it did not work.

It works only if the property is set in hive-site.xml before starting HS2, OR set the env var HIVE_AUX_JARS_PATH to a dir containing my jars OR start HS2 with --hiveconf hive.aux.jars.path=… and it then becomes a global setting. I would like it to become a session specific setting as setting it through a global property for the process needs HS2 restart through admin, and looks like not favored by many of our users.

Is this the way the property is supposed to work or I am doing something wrong?

I need this for our storage handlers, serde, and UDFs to work. Please note that even the “add jar” does not help here.

See Hive ML for full question details

1 ACCEPTED SOLUTION

avatar

One thing to remember here is that Beeline is just a custom JDBC client that is sending the queries to HiveServer2, typically the beeline is sitting on a different machine than the HiveServer2 machine so using local filesytsem paths will not work as they may not be valid on HiveServer2 machine. So to include additional jars in the classpath, first push the jars into hdfs and then use "add jar hdfs://<hdfs_location_of_your_jar>". This should do it.

View solution in original post

9 REPLIES 9

avatar

Ali,

The best solution I found for custom serdes and UDFs is:

  1. create a directory: /usr/hdp/current/hive-server2/auxlib on all hiveserver2 nodes
  2. copy your .jar files to the new folder on all hiveserver2 nodes
  3. restart hiveserver2

No need to change any settings.

avatar

Thanks Guilherme. Am checking with them but not sure it will work as they are looking for a way to avoid sysadmin intervention

avatar
Explorer

Guilherme's solution worked for me on the HDP 2.5 sandbox VM when I used the hive-client directory instead of the hive-server2 directory.

Create directory /usr/hdp/current/hive-client/auxlib/

Move jar files to that directory

Restart Hive

avatar

One thing to remember here is that Beeline is just a custom JDBC client that is sending the queries to HiveServer2, typically the beeline is sitting on a different machine than the HiveServer2 machine so using local filesytsem paths will not work as they may not be valid on HiveServer2 machine. So to include additional jars in the classpath, first push the jars into hdfs and then use "add jar hdfs://<hdfs_location_of_your_jar>". This should do it.

avatar

This option (dropping jars in /usr/hdp/current/hive-server2/auxlib) may not work for them because they have about 800 jars, they in turn load shared libs.The way they currently manage this is by using an uber jar whose manifest’s Class-Path entry has references to relative paths of our jars.The relative paths work because the uber jar resides in one of their own installation directories,which wont happen when the uber jar is in cluster’s installation directory.Copying so many jars to cluster installation will be impractical for admins of joint customers.Is there no way to use ADD JAR or set

avatar
Master Collaborator

Hi: I have the same problem, but still doesnt work:

I have

[bigotes@lnxbig05 auxlib]$ pwd
/usr/hdp/2.3.2.0-2950/hive-server2/auxlib  elasticsearch-hadoop-2.2.0-rc1.jar

1734-sin-titulo.png

and when i run this query i get this error

REATE EXTERNAL TABLE rsi(name STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource'='radio/artists');
Error occurred executing hive query: Error while compiling statement: FAILED: SemanticException Cannot find class 'org.elasticsearch.hadoop.hive.EsStorageHandler'

I don know why Hive doesnt find this class on the jar.

Thanks

avatar
Master Mentor

@Roberto Sancho try to add it in beeline manually

avatar
Master Collaborator

Hi:

I have resolved this from Ambari Hive config interfaz!!

I have added this to the script hive-env.sh

HIVE_AUX_JARS_PATH=/usr/hdp/lib/elasticsearch-hadoop-2.2.0-rc1.jar

And i restarted hive services.

***I always use hive to run the queries, is better beeline????

Many thanks

avatar
Master Mentor

yes use beeline absolutely, hive shell is deprecated and will be removed in the near future due to concurrency and limited security features. @Roberto Sancho