I am badly stuck setting the execution engine to spark and running hive queries on CM 7.03.As i understood Mapreduce as execution engine isn't anymore supported and we have TEZ and SPARK available.
I have tried configuring spark as my execution engine as TEZ looked a bit complex.
Can anyone help please,i am stuck since 3 days
I am using set hive.execution.engine=spark;
and have alos checked the below parameters and values are as below.
Spark Executor Memory + Overhead = 1 gb
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
20/04/02 01:12:29 [HiveServer2-Background-Pool: Thread-47]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
Hi @SwasBigData ,
Thanks for reaching out to Cloudera community. I searched the public doc and found this snippet from the doc:
Cloudera Bug: OPSAPS-52115: Hive service should not depend on Spark starting with C7Hive on Spark related parameters will not appear in the Hive service configuration.
My reading from above is that Hive on Spark may not be supported anymore in C7.x. I would suggest you starting to think to use Hive on Tez.
Thanks and hope this helps!
thanks for getting back,are we saving the only option available for Hive runtime engine is Hive on Tez?As MR is also not supported?
I didn't find enough documentation that could guide me with Hive on Tez configuratiion on CM,if you know any good resources i can follow do advise