Support Questions

Find answers, ask questions, and share your expertise

Hive queries to run on Spark2

New Contributor

Hi Guys,

I have cloudera setuo with following version's

Spark2  version-2.1.0.cloudera2

Hadoop  -2.6.0-cdh5.15.0

Hive release with parcels -1.1.0-cdh5.15.0



I want to run hive queries in my spark application.

In scala it would be like

val df =  sqlContext.sql("Select * from default.test")


. However it alwasy gives me error of database ot view not found, which actually exists.


I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.

I tried copying all hive jar's to spark2/jars/ folder but still same results

Hive Execution engine is MR.


Do i need to add any extra jar or classpath to spark configurations.?


How will spark pick hive jar's. Please help me out on this





Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link

New Contributor


Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.

However when i choose hive it shows all the databases in hive.

Also spark-defaults.conf is got updated with hive-metastore version and hive jars.

But i got a new error of

java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty


 at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(

This error seems incompatible hive.serde jar .

What's the workaround for it ?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.