question Re: Can I use SparkSQL on a cluster using Hive on Spark? in Support Questions

Can I use SparkSQL on a cluster using Hive on Spark?

avengers — Wed, 06 Nov 2019 06:36:02 GMT

I am using CDH 6.1.1 Cluster.

Cluster is configured to use Spark as the execution engine for Hive.

Is there anything wrong with using SparkSQL on this Cluster?

Is it ok to create Hive Tables and change data using SparkSQL?

Since SparkSQL uses the Hive Metastore, I suspect that there may be a conflict between SparkSQL and Hive on Spark.

In addition, please refer to documentation on how to intergrate Cloudera CDH Hive with Apache Zeppelin's Spark interpreter.

Thank you.

Re: Can I use SparkSQL on a cluster using Hive on Spark?

helmi_khalifa — Wed, 06 Nov 2019 10:08:58 GMT

Hi @av ,

Here the links for the Hive and Spark interpreter doc's :

https://zeppelin.apache.org/docs/0.8.2/interpreter/hive.html

https://zeppelin.apache.org/docs/0.8.2/interpreter/spark.html

Best,

Helmi KHALIFA

Re: Can I use SparkSQL on a cluster using Hive on Spark?

avengers — Thu, 07 Nov 2019 06:19:33 GMT

Thanks. However, I have already read them.

I'am already connecting to Hive from Zeppelin using JDBC.

I want to query Hive Table with SparkSQL.

And I'm wondering if the metastore won't crash if I use it in a Cluster using HiveOnSpark.

For example.

%spark val df = spark.read.format("csv").option("header", "true") .option("inferSchema", "true").load("/somefile.csv") df.createOrReplaceTempView("csvTable"); %spark.sql select * from csvTable lt join hiveTable rt on lt.col = rt.col

Re: Can I use SparkSQL on a cluster using Hive on Spark?

helmi_khalifa — Fri, 08 Nov 2019 16:55:10 GMT

Hi @avengers ,

U will need to share variables between two zeppelin interpreters and i dont think that we can do it between spark and sparkSQL.

I find an easier way by using sqlContext inside the same interpreter %spark:

%spark

val df = spark.read.format("csv").option("header", "true")
.option("inferSchema", "true").load("/somefile.csv")

df.createOrReplaceTempView("csvTable");

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

val resultat = sqlContext.sql("select * from csvTable lt join hiveTable rt on lt.col = rt.col")

resultat.show()

I tried it and it works !

Best,

Helmi KHALIFA

Re: Can I use SparkSQL on a cluster using Hive on Spark?

helmi_khalifa — Thu, 14 Nov 2019 10:42:29 GMT

hi @avengers

If it works for you, would you be kind enough to accept the answer please ?

Best,

Helmi KHALIFA

Re: Can I use SparkSQL on a cluster using Hive on Spark?

gsthina — Thu, 14 Nov 2019 10:54:24 GMT

Hey @avengers,

Just thought, this could add some more value to this question here.

Spark SQL uses a Hive Metastore to manage the metadata of persistent relational entities (e.g. databases, tables, columns, partitions) in a relational database (for fast access) [1].

Also, I don't think there would be a MetaStore crash if we use it along with HiveOnSpark.

[1] https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-hive-metastore.html