question Re: Hive cannot see jar in Archives of Support Questions (Read Only)

Hive cannot see jar

ed_day — Wed, 08 Feb 2017 23:03:44 GMT

I have a Hive table tweets stored as text that I am trying to write to another table tweetsORC that is ORC.

I am trying to insert from tweets to tweetsORC using:

INSERT OVERWRITE TABLE tweetsORC SELECT <fields> FROM tweets;

When Hive was started I used:

HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar

this errors with:

File does not exist: /home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar

I have copied the file to that location for all nodes. I have copied it to /usr/hdp/current/hive-server2/auxlib on the Hiveserver2 machine. Still get the error.

I have tried using:

HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar

which gives:

RuntimeException: java.lang.IllegalArgumentException: Wrong FS:

I have tried changing hive.metastore.warehouse.dir to hdfs:///master.royble.co.uk/user/hive/warehouse.

Still get the error.

I'm tearing my hair out! TIA!!

Re: Hive cannot see jar

awoolford — Wed, 08 Feb 2017 23:22:50 GMT

@ed day: if you're trying to access files on the local file system, try adding the `file://` protocol to the path, e.g. `file:///home/ed/...`.

Re: Hive cannot see jar

icocio — Wed, 08 Feb 2017 23:26:55 GMT

@ed day

Have you tried adding the jar in the script?

add jar /path/to/jar/file.jar

Re: Hive cannot see jar

dineshc — Fri, 10 Feb 2017 02:39:38 GMT

@ed day

If you do not specify the file system in the jar path, it will take the local file system path as shown below:

hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar

gets translated to :

hive.aux.jars.path=file:///home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar

In your case, you can try by explicitly adding file:// before the jar path.

If your jar is in the HDFS then use:

hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar

P.S. please verify that the hadoop user you are using to execute these, has the read privileges to your local jar path.

Re: Hive cannot see jar

ed_day — Fri, 10 Feb 2017 17:30:01 GMT

Thanks that solved it!

Re: Hive cannot see jar

ed_day — Fri, 10 Feb 2017 17:31:10 GMT

Those were what I tried. However I think you are right about the user/permissions thing. If I have time I'll see if it helps. Thasnk!