I have a Hive table tweets stored as text that I am trying to write to another table tweetsORC that is ORC.
I am trying to insert from tweets to tweetsORC using:
INSERT OVERWRITE TABLE tweetsORC SELECT <fields> FROM tweets;
When Hive was started I used:
HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=/home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar
this errors with:
File does not exist: /home/ed/Downloads/serde/json-serde-1.3.7-jar-with-dependencies.jar
I have copied the file to that location for all nodes. I have copied it to /usr/hdp/current/hive-server2/auxlib on the Hiveserver2 machine. Still get the error.
I have tried using:
HADOOP_USER_NAME=hdfs hive ... -hiveconf hive.aux.jars.path=hdfs:///master.royble.co.uk/jars/json-serde-1.3.7-jar-with-dependencies.jar
RuntimeException: java.lang.IllegalArgumentException: Wrong FS:
I have tried changing hive.metastore.warehouse.dir to hdfs:///master.royble.co.uk/user/hive/warehouse.
Still get the error.
I'm tearing my hair out! TIA!!
gets translated to :
In your case, you can try by explicitly adding file:// before the jar path.
If your jar is in the HDFS then use:
P.S. please verify that the hadoop user you are using to execute these, has the read privileges to your local jar path.
Those were what I tried. However I think you are right about the user/permissions thing. If I have time I'll see if it helps. Thasnk!