question Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ? in Support Questions

How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Tue, 26 Sep 2017 20:46:46 GMT

I added spark.sql.warehouse.dir in the configuration but still spark is not using it. it still points to /apps/hive/warehouse. I even changed the settings in hive-site.xml but I am still having the same issue. I keep getting this exception

org.apache.hadoop.ipc.RemoteException: Permission denied: user=user, access=WRITE, inode="/apps/hive/warehouse/position_parquet_test/_temporary/0":hive:hdfs:drwxr-xr-x

update:

once i start spark shell it shows in the info that the warehouse was set to the right path

17/09/27 13:16:46 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /uat/06295/app/XTA0/hivedb

how ever when I try to create a table i get the exception mentioned above

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

deepesh1 — Wed, 27 Sep 2017 00:30:08 GMT

The property you want to change is "hive.metastore.warehouse.dir".

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Wed, 27 Sep 2017 00:35:01 GMT

I am using spark 2.1.1 and hive.metastore.warehouse.dir is deprecated on version 2.1.1

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

asirna — Wed, 27 Sep 2017 01:37:40 GMT

Hi @Akrem Latiwesh,

Try setting spark.sql.warehouse.dir in spark conf.

Thanks,

Aditya

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Wed, 27 Sep 2017 02:00:37 GMT

Hi @Aditya Sirna

I am including it in spark conf as the following

spark.sql.warehouse.dir=hdfs:///uat/06295/app/XTA0/hivedb

spark.yarn.security.credentials.hive.enabled=true

But spark it does not use it as hive warehouse

Thanks

Akrem

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

asirna — Wed, 27 Sep 2017 02:14:14 GMT

Hi @Akrem Latiwesh,

Can you just try giving "/app/XTA0/hivedb" instead of hdfs:/// . It will pick the defaultFS from hadoop's core-site.xml.

Thanks,

Aditya

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Wed, 27 Sep 2017 03:10:37 GMT

Thanks @Aditya Sirna

I tried that and I am still getting the same exception

Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: user=USER, access=WRITE, inode="/apps/hive/warehouse/position_parquet_test/_temporary/0":hive:hdfs:drwxr-xr-x

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

deepesh1 — Wed, 27 Sep 2017 04:12:33 GMT

Have you given that a try, setting hive.metastore.warehouse.dir in hive-site.xml under spark conf.

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Thu, 28 Sep 2017 01:34:40 GMT

yes I did. Please look at the log below

17/09/27 14:30:49 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/uat/06295/app/XTA0/hivedb'). 17/09/27 14:30:49 INFO SharedState: Warehouse path is '/uat/06295/app/XTA0/hivedb'. 17/09/27 14:30:49 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3d3238e4{/SQL,null,AVAILABLE,@Spark} 17/09/27 14:30:49 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@53cfe9e5{/SQL/json,null,AVAILABLE,@Spark} 17/09/27 14:30:49 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@75437b4b{/SQL/execution,null,AVAILABLE,@Spark} 17/09/27 14:30:49 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@7e94c1d5{/SQL/execution/json,null,AVAILABLE,@Spark} 17/09/27 14:30:49 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@51b0451f{/static/sql,null,AVAILABLE,@Spark} 17/09/27 14:30:49 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes. 17/09/27 14:30:49 INFO HiveClientImpl: Attempting to login to Kerberos using principal: and keytab: 17/09/27 14:30:49 INFO UserGroupInformation: Login successful for user using keytab file 17/09/27 14:30:50 INFO metastore: Trying to connect to metastore with URI 17/09/27 14:30:50 INFO metastore: Connected to metastore. 17/09/27 14:30:53 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 17/09/27 14:30:53 INFO SessionState: Created local directory: /tmp/369e4278-1d5e-46e6-bc5d-b7143e48c525_resources 17/09/27 14:30:53 INFO SessionState: Created HDFS directory: /tmp/hive/ 17/09/27 14:30:53 INFO SessionState: Created local directory: /tmp/ 17/09/27 14:30:53 INFO SessionState: Created HDFS directory: /tmp/hive/ 17/09/27 14:30:53 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is /uat/06295/app/XTA0/hivedb 17/09/27 14:30:53 INFO Main: Created Spark session with Hive support

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

akrem_latiwesh — Thu, 05 Oct 2017 02:08:01 GMT

The issue has been resolved, I was creating a table without specifying in which database i want to create it. So, hive was trying to create the table in the default database to which i do not have permissions to write.

All what I needed to do is to create a new database and use the command " USE database_name" before i actually create my table.

thanks for the help guys, wasted too much time on configurations while the solution was done by a simple query.

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

mwol — Wed, 15 Jan 2020 15:39:00 GMT

Although this thread is quite old, I post my solution, as it may still be helpful to some people.
I encountered the problem in spark2-shell, as the hive metastore.uri was not set in client mode,

After adding the following to spark-defaults.conf (via cloudera manger's

Spark 2 Client Advanced Configuration Snippet (Safety Valve) for spark2-conf/spark-defaults.conf)

spark.sql.warehouse.dir=hdfs://nameservice1/user/hive/warehouse
spark.executor.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083
spark.driver.extraJavaOptions=-Dhive.metastore.uris=thrift://hive.metastore.net:9083

I could list and access all tables from the hive metastore from within spark-shell.

Re: How can I change hive warehouse directory from /apps/hive/warehouse/ to another warehouse that i have permission to write to ?

mwol — Wed, 15 Jan 2020 15:40:30 GMT

... you have to adapt the configuration to your needs, of course, by changing metastore host and warehouse dir, accordingly