10-08-2018
02:37 AM
- last edited on
10-08-2018
09:07 AM
by
cjervis
Hi Guys,
I have cloudera setuo with following version's
Spark2 version-2.1.0.cloudera2
Hadoop -2.6.0-cdh5.15.0
Hive release with parcels -1.1.0-cdh5.15.0
I want to run hive queries in my spark application.
In scala it would be like
val df = sqlContext.sql("Select * from default.test")
df.show()
. However it alwasy gives me error of database ot view not found, which actually exists.
I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.
I tried copying all hive jar's to spark2/jars/ folder but still same results
Hive Execution engine is MR.
Do i need to add any extra jar or classpath to spark configurations.?
How will spark pick hive jar's. Please help me out on this
10-08-2018 07:47 PM - edited 10-08-2018 07:48 PM
Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link
10-09-2018 03:00 AM
Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.
However when i choose hive it shows all the databases in hive.
Also spark-defaults.conf is got updated with hive-metastore version and hive jars.
But i got a new error of
java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty
at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:99)
This error seems incompatible hive.serde jar .
What's the workaround for it ?