Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Trouble creating databases in Hive using Spark 2.0

Hi Team,

I am trying to create a database in Hive using spark sql in spark 2.0 version & encounter the below permission issue,

command - spark.sql("create database test") ************************************************************************************************************************

Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Permission denied: ************************************************************************************************************************

However when I try to do the same in Spark 1.6 version, I am able to do it without any issues. command - sqlContext.sql("create database test")

Do I need to set some configuration to execute from Spark 2.0?

Thanks in Advance!


Hi @Rohit Ravishankar,

spark 1.6 and spark 2 have a different hive warehouse.

For spark 1.6 , warehouse location is identified by hive.metastore.warehouse.dir (default = /apps/hive/warehouse)

For spark 2, warehouse location is identified by spark.sql.warehouse.dir (default = <user.dir>/spark-warehouse)

I guess the user has right permissions to spark 1.6's warehouse location but not spark 2's location. Can you please check it and try giving proper permissions and run the query.



Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.