About dsnospam1

dsnospam1 · ‎05-18-2018

@Felix Albani, Felix Albani I would be glad to mark the answer as helpful, but don't know how to do that.

dsnospam1 · ‎05-18-2018

@Felix Albani I would be glad to mark the answer as helpful, but don't know how to do that.

dsnospam1 · ‎05-17-2018

Hi @Felix Albani, According to @Aditya Sirna's reply to a similar thread, Spark 2 (which is what I am using - NOT Spark 1) has a different warehouse location, which, I suppose, explains why LOCATION must be used. @Aditya Sirna, if I want to create a Hive database with Spark, do I have to use the location statement? If so, what location statement should I use if I want to keep my databases and tables managed by the Hive metastore?

dsnospam1 · ‎05-17-2018

According to this Hortonworks community URL, Location is NOT mandatory. But it was the only way I was able to create a database.

dsnospam1 · ‎05-17-2018

@Felix Albani Thank you for your reply. That suggestion actually worked! However, I don't understand why it is necessary to specify the database location in HDFS. Why does that have to be done in HDP? In other Hadoop/Spark distributions, I haven't had to specify the database filepath and database name when creating Hive databases with Spark. I still believe there is a configuration problem with Hive and Spark with HDP.

dsnospam1 · ‎05-16-2018

Hi @Felix Albani, Thanks for your reply. Unfortunately, the suggestion didn't work. First, it took FOREVER to launch pyspark with the yarn option $ pyspark --master yarn option (and I still don't understand why that option was needed). And also, when it did launch, it ultimately threw a bunch of java errors.

dsnospam1 · ‎05-16-2018

@Felix_Albani, Thanks for your reply. Unfortunately, the suggestion didn't work. First, it took FOREVER to launch pyspark with the yarn option $ pyspark --master yarn option (and I still don't understand why that option was needed). And also, when it did launch, it ultimately threw a bunch of java errors.

dsnospam1 · ‎05-16-2018

Hi, After installing HDP 2.6.3, I ran Pyspark in the terminal, then initiated a Spark Session, and tried to create a new database (see last line of code: $ pyspark > from pyspark.sql import SparkSession > spark = SparkSession.builder.master("local").appName("test").enableHiveSupport().getOrCreate() > spark.sql("show databases").show() > spark.sql("create database if not exists NEW_DB") However, PySpark threw an error where it was trying to create a database locally: AnalysisException: 'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to create database path file:/home/jdoe/spark-warehouse/new_db.db, failed to create database new_db);' I wasn't trying to create a database locally. I was trying to create a database within Hive. Is there a configuration problem with HDP 2.6.3? Please advise. Thanks.

Online	Offline
Last Visited	‎07-09-2018 08:53 PM

Member Since	‎04-23-2018 02:23 PM
Last Visited	‎07-09-2018 08:53 PM
Posts	20

Cloudera Community

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

Re: HDP 2.6 Spark can't create database - configur...

HDP 2.6 Spark can't create database - configuratio...