Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Livy cannot find the database

Solved Go to solution

Livy cannot find the database

Expert Contributor

Hi,

 

I switched to Spark 2.2 and latest version of Livy and now I am having problem with Hive.

 

For instance this works in spark2-shell:

 

scala> spark.sql("select * from bikeshare.trips")
res5: org.apache.spark.sql.DataFrame = [tripid: int, duration: int ... 9 more fields]

 

But the same gives me this error in Hue notebook (Spark2.2)

 

In Zeppelin

spark.sql("select * from bikeshare.trips")
org.apache.spark.sql.AnalysisException: Table or view not found: `bikeshare`.`trips`; line 1 pos 14;

 

In Hue

spark.sql("select * from bikshare.trips")

'Project [*] +- 'UnresolvedRelation `bikshare`.`trips` at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:82) at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:78) at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126) at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126) at scala.collection.immutable.List.foreach(List.scala:381) at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:78) at
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:91) at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:52) at
org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:66) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:623) ... 47 elided

 

The spark.catalog shows empty table

 

spark.catalog.listTables.show

+----+--------+-----------+---------+-----------+
|name|database|description|tableType|isTemporary|
+----+--------+-----------+---------+-----------+
+----+--------+-----------+---------+-----------+

Listing databases in spark-shell displays all the databases but in Hue/Livy

 

spark.sql("show databases").show
+------------+
|databaseName|
+------------+
|     default|
+------------+

So of course the bikeshare doesn't exist from Livy

spark.sql("use bikshare")
org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'bikshare' not found;

Is there something I am missing here? It used to work in spark 1.6 wih sqlContext. It's like it couldn't load metastore metadata about all the warehouse directories etc.

 

Update: Hive Service for Spark 2 is enabled

 

Many thanks,

Maziyar

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Livy cannot find the database

Expert Contributor

I had to link hive-site.xml to spark conf dir manually!

 

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf

 

sudo ln -s /etc/hive/conf/hive-site.xml $SPARK_CONF_DIR

 I am not sure why this is not already in spark conf.

 

Problem has been solved.

2 REPLIES 2

Re: Livy cannot find the database

Expert Contributor

I had to link hive-site.xml to spark conf dir manually!

 

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf

 

sudo ln -s /etc/hive/conf/hive-site.xml $SPARK_CONF_DIR

 I am not sure why this is not already in spark conf.

 

Problem has been solved.

Highlighted

Re: Livy cannot find the database

New Contributor

I had the simillar issue, although symlinking did not help.

 

What I did was setting

livy.repl.enableHiveContext = true

in my livy.conf file. It is commented by default.