Hive on Spark Cloudera Manager 7.03 Unable to Proceed

[#1]

Hi All

I am badly stuck  setting the execution engine to spark and running hive queries on CM 7.03.As i understood Mapreduce  as execution engine isn't anymore supported and we have TEZ and SPARK available.

I have tried configuring spark as my execution engine as TEZ looked a bit complex.

Can anyone help please,i am stuck since 3 days

I  am using set hive.execution.engine=spark; 

and have alos checked the below parameters and values are as below.

yarn.scheduler.maximum-allocation-mb=10g

yarn.nodemanager.resource.memory-mb=40g

hive.spark.client.server.connect.timeout=90000ms

hive.spark.client.connect.timeout=90000ms

Spark Executor Memory + Overhead = 1 gb

**********************

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
20/04/02 01:12:29 [HiveServer2-Background-Pool: Thread-47]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'

Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

[#2]

Any info on the above issue,please update

[#3]

Hi @SwasBigData ,

Thanks for reaching out to Cloudera community. I searched the public doc and found this snippet from the doc[1]:

Cloudera Bug: OPSAPS-52115: Hive service should not depend on Spark starting with C7Hive on Spark related parameters will not appear in the Hive service configuration.

My reading from above is that Hive on Spark may not be supported anymore in C7.x. I would suggest you starting to think to use Hive on Tez.

Thanks and hope this helps!

Li

[1]: https://docs.cloudera.com/cloudera-manager/7.1.0/release-notes/topics/cm-fixed-issues-7.1.0.html

[#4]

Hello Li

thanks for getting back,are we saving the only option available for Hive  runtime engine is Hive on Tez?As MR is also not supported?

I didn't find enough documentation that could guide me with Hive on Tez configuratiion on CM,if you know any good resources i can follow do advise

thank you