Support Questions

Find answers, ask questions, and share your expertise

How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

avatar

I added spark.sql.warehouse.dir in the configuration but still spark is not using it. it still points to /apps/hive/warehouse. I even changed the settings in hive-site.xml but I am still having the same issue. I keep getting this exception

org.apache.hadoop.ipc.RemoteException: Permission denied: user=user, access=WRITE, inode="/apps/hive/warehouse/position_parquet_test/_temporary/0":hive:hdfs:drwxr-xr-x

update:

once i start spark shell it shows in the info that the warehouse was set to the right path

17/09/27 13:16:46 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /uat/06295/app/XTA0/hivedb

how ever when I try to create a table i get the exception mentioned above

11 REPLIES 11

avatar
Explorer

Although this thread is quite old, I post my solution, as it may still be helpful to some people.
I encountered the problem in spark2-shell, as the hive metastore.uri was not set in client mode,

 

After adding the following to spark-defaults.conf (via cloudera manger's

Spark 2 Client Advanced Configuration Snippet (Safety Valve) for spark2-conf/spark-defaults.conf)

 

 

 

spark.sql.warehouse.dir=hdfs://nameservice1/user/hive/warehouse
spark.executor.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083
spark.driver.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083

 

 I could list and access all tables from the hive metastore from within spark-shell.

avatar
Explorer

... you have to adapt the configuration to your needs, of course, by changing metastore host and warehouse dir, accordingly