I have a Hive table tweets stored as text that I am trying to write to another table tweetsORC that is ORC.
I am trying to insert from tweets to tweetsORC using:
INSERT OVERWRITE TABLE tweetsORC SELECT <fields> FROM tweets;
When Hive was started I used:
HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar
this errors with:
File does not exist: /home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar
I have copied the file to that location for all nodes. I have copied it to /usr/hdp/current/hive-server2/auxlib on the Hiveserver2 machine. Still get the error.
I have tried using:
HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar
RuntimeException: java.lang.IllegalArgumentException: Wrong FS:
I have tried changing hive.metastore.warehouse.dir to hdfs:///master.royble.co.uk/user/hive/warehouse.
Still get the error.
I'm tearing my hair out! TIA!!
Have you tried adding the jar in the script?
add jar /path/to/jar/file.jar
View solution in original post
@ed day: if you're trying to access files on the local file system, try adding the `file://` protocol to the path, e.g. `file:///home/ed/...`.
Thanks that solved it!
If you do not specify the file system in the jar path, it will take the local file system path as shown below:
gets translated to :
In your case, you can try by explicitly adding file:// before the jar path.
If your jar is in the HDFS then use:
P.S. please verify that the hadoop user you are using to execute these, has the read privileges to your local jar path.
Those were what I tried. However I think you are right about the user/permissions thing. If I have time I'll see if it helps. Thasnk!