Created on
04-01-2020
11:32 PM
- last edited on
04-02-2020
12:32 AM
by
VidyaSargur
Hi All
I am badly stuck setting the execution engine to spark and running hive queries on CM 7.03.As i understood Mapreduce as execution engine isn't anymore supported and we have TEZ and SPARK available.
I have tried configuring spark as my execution engine as TEZ looked a bit complex.
Can anyone help please,i am stuck since 3 days
I am using set hive.execution.engine=spark;
and have alos checked the below parameters and values are as below.
yarn.scheduler.maximum-allocation-mb=10g
yarn.nodemanager.resource.memory-mb=40g
hive.spark.client.server.connect.timeout=90000ms
hive.spark.client.connect.timeout=90000ms
Spark Executor Memory + Overhead = 1 gb
**********************
Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
20/04/02 01:12:29 [HiveServer2-Background-Pool: Thread-47]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
Created 04-03-2020 07:25 AM
Any info on the above issue,please update
Created 04-03-2020 10:23 AM
Hi @SwasBigData ,
Thanks for reaching out to Cloudera community. I searched the public doc and found this snippet from the doc[1]:
Cloudera Bug: OPSAPS-52115: Hive service should not depend on Spark starting with C7Hive on Spark related parameters will not appear in the Hive service configuration.
My reading from above is that Hive on Spark may not be supported anymore in C7.x. I would suggest you starting to think to use Hive on Tez.
Thanks and hope this helps!
Li
[1]: https://docs.cloudera.com/cloudera-manager/7.1.0/release-notes/topics/cm-fixed-issues-7.1.0.html
Li Wang, Technical Solution Manager
Created 04-03-2020 10:42 PM
Hello Li
thanks for getting back,are we saving the only option available for Hive runtime engine is Hive on Tez?As MR is also not supported?
I didn't find enough documentation that could guide me with Hive on Tez configuratiion on CM,if you know any good resources i can follow do advise
thank you