10-08-2018 02:37 AM - last edited on 10-08-2018 09:07 AM by cjervis
I have cloudera setuo with following version's
Hive release with parcels -1.1.0-cdh5.15.0
I want to run hive queries in my spark application.
In scala it would be like
val df = sqlContext.sql("Select * from default.test")
. However it alwasy gives me error of database ot view not found, which actually exists.
I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.
I tried copying all hive jar's to spark2/jars/ folder but still same results
Hive Execution engine is MR.
Do i need to add any extra jar or classpath to spark configurations.?
How will spark pick hive jar's. Please help me out on this
10-08-2018 07:47 PM - edited 10-08-2018 07:48 PM
Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link
10-09-2018 03:00 AM
Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.
However when i choose hive it shows all the databases in hive.
Also spark-defaults.conf is got updated with hive-metastore version and hive jars.
But i got a new error of
This error seems incompatible hive.serde jar .
What's the workaround for it ?