Hive cannot see jar

[#1]

I have a Hive table tweets stored as text that I am trying to write to another table tweetsORC that is ORC.

I am trying to insert from tweets to tweetsORC using:

INSERT OVERWRITE TABLE tweetsORC SELECT <fields> FROM tweets;

When Hive was started I used:

HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar 

this errors with:

File does not exist: /home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar

I have copied the file to that location for all nodes. I have copied it to /usr/hdp/current/hive-server2/auxlib on the Hiveserver2 machine. Still get the error.

I have tried using:

HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar

which gives:

RuntimeException: java.lang.IllegalArgumentException: Wrong FS: 

I have tried changing hive.metastore.warehouse.dir to hdfs:///master.royble.co.uk/user/hive/warehouse.

Still get the error.

I'm tearing my hair out! TIA!!

[#2]

@ed day: if you're trying to access files on the local file system, try adding the `file://` protocol to the path, e.g. `file:///home/ed/...`.

[#3 · Accepted solution]

@ed day

Have you tried adding the jar in the script?

add jar /path/to/jar/file.jar

[#4]

Thanks that solved it!

[#5]

@ed day

If you do not specify the file system in the jar path, it will take the local file system path as shown below:

hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar 

gets translated to :

hive.aux.jars.path=file:///home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar 

In your case, you can try by explicitly adding file:// before the jar path.

If your jar is in the HDFS then use:

hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar	

P.S. please verify that the hadoop user you are using to execute these, has the read privileges to your local jar path.

[#6]

Those were what I tried. However I think you are right about the user/permissions thing. If I have time I'll see if it helps. Thasnk!