Reply
New Contributor
Posts: 2
Registered: ‎10-08-2018

Hive queries to run on Spark2

[ Edited ]

Hi Guys,

I have cloudera setuo with following version's

Spark2  version-2.1.0.cloudera2

Hadoop  -2.6.0-cdh5.15.0

Hive release with parcels -1.1.0-cdh5.15.0

 

 

I want to run hive queries in my spark application.

In scala it would be like

val df =  sqlContext.sql("Select * from default.test")

df.show()

 

. However it alwasy gives me error of database ot view not found, which actually exists.

 

I added hive lib directory to spark.driver.extraClassPath in spark_defaults.conf. But still same results.

I tried copying all hive jar's to spark2/jars/ folder but still same results

Hive Execution engine is MR.

 

Do i need to add any extra jar or classpath to spark configurations.?

 

How will spark pick hive jar's. Please help me out on this

Posts: 508
Topics: 14
Kudos: 85
Solutions: 45
Registered: ‎09-02-2016

Re: Hive queries to run on Spark2

[ Edited ]

@kush

 

Please make sure "hive-site.xml" is present in /etc/spark/conf (or) in the corresponding spark2 config directory. You can either copy paste from hive config directory (or) create a soft link

New Contributor
Posts: 2
Registered: ‎10-08-2018

Re: Hive queries to run on Spark2

@saranvisa

Thanks for your reply. I already copied it but it seems we need to choose hive as option in hive-service configurations in cloudera manager.

However when i choose hive it shows all the databases in hive.

Also spark-defaults.conf is got updated with hive-metastore version and hive jars.

But i got a new error of

java.lang.NoSuchMethodError: org.apache.hive.common.util.HiveStringUtils.joinIgnoringEmpty

 

 at org.apache.hadoop.hive.serde2.ColumnProjectionUtils.appendReadColumns(ColumnProjectionUtils.java:99)

This error seems incompatible hive.serde jar .

What's the workaround for it ?

Announcements