Created 09-26-2017 01:46 PM
I added spark.sql.warehouse.dir in the configuration but still spark is not using it. it still points to /apps/hive/warehouse. I even changed the settings in hive-site.xml but I am still having the same issue. I keep getting this exception
org.apache.hadoop.ipc.RemoteException: Permission denied: user=user, access=WRITE, inode="/apps/hive/warehouse/position_parquet_test/_temporary/0":hive:hdfs:drwxr-xr-x
update:
once i start spark shell it shows in the info that the warehouse was set to the right path
17/09/27 13:16:46 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /uat/06295/app/XTA0/hivedb
how ever when I try to create a table i get the exception mentioned above
Created 01-15-2020 07:39 AM
Although this thread is quite old, I post my solution, as it may still be helpful to some people.
I encountered the problem in spark2-shell, as the hive metastore.uri was not set in client mode,
After adding the following to spark-defaults.conf (via cloudera manger's
spark.sql.warehouse.dir=hdfs://nameservice1/user/hive/warehouse
spark.executor.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083
spark.driver.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083
I could list and access all tables from the hive metastore from within spark-shell.
Created 01-15-2020 07:40 AM
... you have to adapt the configuration to your needs, of course, by changing metastore host and warehouse dir, accordingly