Skip to main content
avengers
avengers
Silver ⭐️⭐️⭐️
November 6, 2019
Solved

Can I use SparkSQL on a cluster using Hive on Spark?

I am using CDH 6.1.1 Cluster.

Cluster is configured to use Spark as the execution engine for Hive.

 

 

Is there anything wrong with using SparkSQL on this Cluster?

Is it ok to create Hive Tables and change data using SparkSQL?

 

Since SparkSQL uses the Hive Metastore, I suspect that there may be a conflict between SparkSQL and Hive on Spark.

 

In addition, please refer to documentation on how to intergrate Cloudera CDH Hive with Apache Zeppelin's Spark interpreter. 

 

Thank you.

 

gsthina
gsthinaAnswer
Gold ⭐️
November 14, 2019

Hey @avengers,

 

Just thought, this could add some more value to this question here.

 

Spark SQL uses a Hive Metastore to manage the metadata of persistent relational entities (e.g. databases, tables, columns, partitions) in a relational database (for fast access) [1].

 

Also, I don't think there would be a MetaStore crash if we use it along with HiveOnSpark.

 

[1] https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-hive-metastore.html

View in thread

5 replies

helmi_khalifa
helmi_khalifa
Gold ⭐️
November 6, 2019

Hi @av_01 ,

 

Here the links for the Hive and Spark interpreter doc's :

https://zeppelin.apache.org/docs/0.8.2/interpreter/hive.html

https://zeppelin.apache.org/docs/0.8.2/interpreter/spark.html

 

Best,

Helmi KHALIFA

 

Translate
avengers
avengersAuthor
Silver ⭐️⭐️⭐️
November 7, 2019

 

Thanks. However, I have already read them.

I'am already connecting to Hive from Zeppelin using JDBC.

 

I want to query Hive Table with SparkSQL.

And I'm wondering if the metastore won't crash if I use it in a Cluster using HiveOnSpark.

 

For example.

 

 

%spark
val df = spark.read.format("csv").option("header", "true")
.option("inferSchema", "true").load("/somefile.csv")

df.createOrReplaceTempView("csvTable");

%spark.sql
select * 
from csvTable lt
join hiveTable rt
on lt.col = rt.col

 

 

Translate
helmi_khalifa
helmi_khalifa
Gold ⭐️
November 8, 2019

Hi @avengers ,

 

U will need to share variables between two zeppelin interpreters and i dont think that we can do it between spark and sparkSQL.

 

I find an easier way by using sqlContext inside the same interpreter %spark:

 

%spark


val df = spark.read.format("csv").option("header", "true")
.option("inferSchema", "true").load("/somefile.csv")

 

df.createOrReplaceTempView("csvTable");

 

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

 

val resultat = sqlContext.sql("select * from csvTable lt join hiveTable rt on lt.col = rt.col")

 

resultat.show()

 

I tried it and it works ! 

 

Best,

Helmi KHALIFA

Translate
helmi_khalifa
helmi_khalifa
Gold ⭐️
November 14, 2019

hi @avengers 

 

If it works for you, would you be kind enough to accept the answer please ?

 

Best,

Helmi KHALIFA

Translate
Terms & ConditionsAccessibility statement