Hive queries to run on Spark2 [ Edited ]

[kush]

Hi Guys,

I have cloudera setuo with following version's

Spark2  version-2.1.0.cloudera2

Hadoop  -2.6.0-cdh5.15.0

Hive release with parcels -1.1.0-cdh5.15.0

I want to run hive queries in my spark application.

In scala it would be like

val df =  sqlContext.sql("Select * from default.test")

df.show()

. However it alwasy gives me error of database ot view not found, which actually exists.

I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.

I tried copying all hive jar's to spark2/jars/ folder but still same results

Hive Execution engine is MR.

Do i need to add any extra jar or classpath to spark configurations.?

How will spark pick hive jar's. Please help me out on this

[saranvisa Champion]

@kush

Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link

[kush]

@saranvisa

Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.

However when i choose hive it shows all the databases in hive.

Also spark-defaults.conf is got updated with hive-metastore version and hive jars.

But i got a new error of

java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty

 at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:99)

This error seems incompatible hive.serde jar .

What's the workaround for it ?