Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive on Spark Cloudera Manager 7.03 Unable to Proceed

Hive on Spark Cloudera Manager 7.03 Unable to Proceed

Explorer

Hi All

 

I am badly stuck  setting the execution engine to spark and running hive queries on CM 7.03.As i understood Mapreduce  as execution engine isn't anymore supported and we have TEZ and SPARK available.

 

I have tried configuring spark as my execution engine as TEZ looked a bit complex.

 

Can anyone help please,i am stuck since 3 days

 

I  am using set hive.execution.engine=spark; 

 

and have alos checked the below parameters and values are as below.

yarn.scheduler.maximum-allocation-mb=10g

yarn.nodemanager.resource.memory-mb=40g

 

hive.spark.client.server.connect.timeout=90000ms

hive.spark.client.connect.timeout=90000ms

 

Spark Executor Memory + Overhead = 1 gb

 

**********************

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'
20/04/02 01:12:29 [HiveServer2-Background-Pool: Thread-47]: ERROR spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 431866dd-3dc5-40c3-b8bc-a6ab1b5883f2)'

Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

 

3 REPLIES 3
Highlighted

Re: Hive on Spark Cloudera Manager 7.03 Unable to Proceed

Explorer

Any info on the above issue,please update

Highlighted

Re: Hive on Spark Cloudera Manager 7.03 Unable to Proceed

Super Collaborator

Hi @SwasBigData ,

 

Thanks for reaching out to Cloudera community. I searched the public doc and found this snippet from the doc[1]:

Cloudera Bug: OPSAPS-52115: Hive service should not depend on Spark starting with C7Hive on Spark related parameters will not appear in the Hive service configuration.

My reading from above is that Hive on Spark may not be supported anymore in C7.x. I would suggest you starting to think to use Hive on Tez.

 

Thanks and hope this helps!

Li

[1]: https://docs.cloudera.com/cloudera-manager/7.1.0/release-notes/topics/cm-fixed-issues-7.1.0.html

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted

Re: Hive on Spark Cloudera Manager 7.03 Unable to Proceed

Explorer

Hello Li

 

thanks for getting back,are we saving the only option available for Hive  runtime engine is Hive on Tez?As MR is also not supported?

 

I didn't find enough documentation that could guide me with Hive on Tez configuratiion on CM,if you know any good resources i can follow do advise

 

thank you

Don't have an account?
Coming from Hortonworks? Activate your account here