(hive) How to dynamically set hive.aux.jars.path environment variable after HiveServer2 is started.

[#1]

Getting the below question in the context of an application that creates tables with a custom storage handler:

When I tried setting jar locations on the fly using “set hive.aux.jars.path=<file urls>” on beeline prompt after connecting to HS2, it did not work.

It works only if the property is set in hive-site.xml before starting HS2, OR set the env var HIVE_AUX_JARS_PATH to a dir containing my jars OR start HS2 with --hiveconf hive.aux.jars.path=… and it then becomes a global setting. I would like it to become a session specific setting as setting it through a global property for the process needs HS2 restart through admin, and looks like not favored by many of our users.

Is this the way the property is supposed to work or I am doing something wrong?

I need this for our storage handlers, serde, and UDFs to work. Please note that even the “add jar” does not help here.

See Hive ML for full question details

[#2]

Ali,

The best solution I found for custom serdes and UDFs is:

1. create a directory: /usr/hdp/current/hive-server2/auxlib on all hiveserver2 nodes
2. copy your .jar files to the new folder on all hiveserver2 nodes
3. restart hiveserver2

No need to change any settings.

[#3]

Thanks Guilherme. Am checking with them but not sure it will work as they are looking for a way to avoid sysadmin intervention

[#4]

Guilherme's solution worked for me on the HDP 2.5 sandbox VM when I used the hive-client directory instead of the hive-server2 directory.

Create directory /usr/hdp/current/hive-client/auxlib/

Move jar files to that directory

Restart Hive

[#5 · Accepted solution]

One thing to remember here is that Beeline is just a custom JDBC client that is sending the queries to HiveServer2, typically the beeline is sitting on a different machine than the HiveServer2 machine so using local filesytsem paths will not work as they may not be valid on HiveServer2 machine. So to include additional jars in the classpath, first push the jars into hdfs and then use "add jar hdfs://<hdfs_location_of_your_jar>". This should do it.

[#6]

This option (dropping jars in /usr/hdp/current/hive-server2/auxlib) may not work for them because they have about 800 jars, they in turn load shared libs.The way they currently manage this is by using an uber jar whose manifest’s Class-Path entry has references to relative paths of our jars.The relative paths work because the uber jar resides in one of their own installation directories,which wont happen when the uber jar is in cluster’s installation directory.Copying so many jars to cluster installation will be impractical for admins of joint customers.Is there no way to use ADD JAR or set

[#7]

Hi: I have the same problem, but still doesnt work:

I have

[bigotes@lnxbig05 auxlib]$ pwd
/usr/hdp/2.3.2.0-2950/hive-server2/auxlib  elasticsearch-hadoop-2.2.0-rc1.jar

and when i run this query i get this error

REATE EXTERNAL TABLE rsi(name STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource'='radio/artists');

Error occurred executing hive query: Error while compiling statement: FAILED: SemanticException Cannot find class 'org.elasticsearch.hadoop.hive.EsStorageHandler'

I don know why Hive doesnt find this class on the jar.

Thanks

[#8]

@Roberto Sancho try to add it in beeline manually

[#9]

Hi:

I have resolved this from Ambari Hive config interfaz!!

I have added this to the script hive-env.sh

HIVE_AUX_JARS_PATH=/usr/hdp/lib/elasticsearch-hadoop-2.2.0-rc1.jar

And i restarted hive services.

***I always use hive to run the queries, is better beeline????

Many thanks

[#10]

yes use beeline absolutely, hive shell is deprecated and will be removed in the near future due to concurrency and limited security features. @Roberto Sancho