Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive queries to run on Spark2

Highlighted

Hive queries to run on Spark2

New Contributor

Hi Guys,

I have cloudera setuo with following version's

Spark2  version-2.1.0.cloudera2

Hadoop  -2.6.0-cdh5.15.0

Hive release with parcels -1.1.0-cdh5.15.0

 

 

I want to run hive queries in my spark application.

In scala it would be like

val df =  sqlContext.sql("Select * from default.test")

df.show()

 

. However it alwasy gives me error of database ot view not found, which actually exists.

 

I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.

I tried copying all hive jar's to spark2/jars/ folder but still same results

Hive Execution engine is MR.

 

Do i need to add any extra jar or classpath to spark configurations.?

 

How will spark pick hive jar's. Please help me out on this

2 REPLIES 2

Re: Hive queries to run on Spark2

Champion

@kush

 

Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link

Re: Hive queries to run on Spark2

New Contributor

@saranvisa

Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.

However when i choose hive it shows all the databases in hive.

Also spark-defaults.conf is got updated with hive-metastore version and hive jars.

But i got a new error of

java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty

 

 at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:99)

This error seems incompatible hive.serde jar .

What's the workaround for it ?