Support Questions

Find answers, ask questions, and share your expertise

Trouble creating databases in Hive using Spark 2.0

Hi Team,

I am trying to create a database in Hive using spark sql in spark 2.0 version & encounter the below permission issue,

command - spark.sql("create database test") ************************************************************************************************************************

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Permission denied: ************************************************************************************************************************

However when I try to do the same in Spark 1.6 version, I am able to do it without any issues. command - sqlContext.sql("create database test")

Do I need to set some configuration to execute from Spark 2.0?

Thanks in Advance!


Hi @Rohit Ravishankar,

spark 1.6 and spark 2 have a different hive warehouse.

For spark 1.6 , warehouse location is identified by hive.metastore.warehouse.dir (default = /apps/hive/warehouse)

For spark 2, warehouse location is identified by spark.sql.warehouse.dir (default = <user.dir>/spark-warehouse)

I guess the user has right permissions to spark 1.6's warehouse location but not spark 2's location. Can you please check it and try giving proper permissions and run the query.