Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can i directly change hive execution engine from tez to spark

Highlighted

can i directly change hive execution engine from tez to spark

I am using HDP-3.0. In previous versions hive.execution.engine was mapreduce/tez. Now in hdp3.0 that selection drop down is not there and default one is tez. some where i read that hive in hdp3.0 support tez(default),llap,spark. Is there any provision to directly set spark as the execution engine.

Thanks & Regards
Rahul

6 REPLIES 6
Highlighted

Re: can i directly change hive execution engine from tez to spark

Super Collaborator

I found this: https://community.hortonworks.com/questions/91550/hive-execution-engine-set-to-spark-is-recommended....

Maybe @Jay Kumar SenSharma can provide a hint if there has something changed on the Spark support in hdp since last year.

Otherwise the documentation on how to set up spark as execution engine for hive is described here:

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

Highlighted

Re: can i directly change hive execution engine from tez to spark

@Rahul P Could you please point me to the doc which says we can use spark in hive as execution engine?

AFAIK we don't have option directly to set spark execution in hive.

What is new is a HiveWarehouseConnector for Spark. Since all the tables in Hive will be transactional form HDP3.0, A HiveWarehouseConnector was introduced in spark to access hive tables.

Ref: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/migrating-data/content/hive_hivewarehouseco...

Highlighted

Re: can i directly change hive execution engine from tez to spark

Dear Sandeep
From this image i thought that deafult is tez and it also support llap& spark
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/hive-overview/concepts/images/hive-sql-quer...

Highlighted

Re: can i directly change hive execution engine from tez to spark

Thanks for pointing to the doc @Rahul P. The image is bit confusing(will try to get this updated). However as i said in my earlier comment Spark on Hive doesn't work out of the box.

Hope this helps.

Highlighted

Re: can i directly change hive execution engine from tez to spark

@Rahul P The doc is now corrected: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/hive-overview/content/hive-apache-hive-3-ar...

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Highlighted

Re: can i directly change hive execution engine from tez to spark

New Contributor

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_s...

Eventually, you set spark as the execution engine?

I've set up the configuration item, but it doesn't work.

errors:

SQL 错误 [40000] [42000]: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819



Don't have an account?
Coming from Hortonworks? Activate your account here