Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive queries to run on Spark2

Hive queries to run on Spark2

New Contributor

Hi Guys,

I have cloudera setuo with following version's

Spark2  version-2.1.0.cloudera2

Hadoop  -2.6.0-cdh5.15.0

Hive release with parcels -1.1.0-cdh5.15.0



I want to run hive queries in my spark application.

In scala it would be like

val df =  sqlContext.sql("Select * from default.test")


. However it alwasy gives me error of database ot view not found, which actually exists.


I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.

I tried copying all hive jar's to spark2/jars/ folder but still same results

Hive Execution engine is MR.


Do i need to add any extra jar or classpath to spark configurations.?


How will spark pick hive jar's. Please help me out on this


Re: Hive queries to run on Spark2




Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link


Re: Hive queries to run on Spark2

New Contributor


Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.

However when i choose hive it shows all the databases in hive.

Also spark-defaults.conf is got updated with hive-metastore version and hive jars.

But i got a new error of

java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty


 at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(

This error seems incompatible hive.serde jar .

What's the workaround for it ?