can i directly change hive execution engine from tez to spark

[#1]

I am using HDP-3.0. In previous versions hive.execution.engine was mapreduce/tez. Now in hdp3.0 that selection drop down is not there and default one is tez. some where i read that hive in hdp3.0 support tez(default),llap,spark. Is there any provision to directly set spark as the execution engine.

Thanks & Regards
Rahul

[#2]

I found this: https://community.hortonworks.com/questions/91550/hive-execution-engine-set-to-spark-is-recommended....

Maybe @Jay Kumar SenSharma can provide a hint if there has something changed on the Spark support in hdp since last year.

Otherwise the documentation on how to set up spark as execution engine for hive is described here:

https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started

[#3]

@Rahul P Could you please point me to the doc which says we can use spark in hive as execution engine?

AFAIK we don't have option directly to set spark execution in hive.

What is new is a HiveWarehouseConnector for Spark. Since all the tables in Hive will be transactional form HDP3.0, A HiveWarehouseConnector was introduced in spark to access hive tables.

Ref: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/migrating-data/content/hive_hivewarehouseco...

[#4]

Dear Sandeep
From this image i thought that deafult is tez and it also support llap& spark
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/hive-overview/concepts/images/hive-sql-quer...

[#5]

Thanks for pointing to the doc @Rahul P. The image is bit confusing(will try to get this updated). However as i said in my earlier comment Spark on Hive doesn't work out of the box.

Hope this helps.

[#6]

@Rahul P The doc is now corrected: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/hive-overview/content/hive-apache-hive-3-ar...

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#7]

https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/integrating-hive/content/hive_configure_a_s...

Eventually, you set spark as the execution engine?

I've set up the configuration item, but it doesn't work.

errors:

SQL 错误 [40000] [42000]: Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

Error while compiling statement: FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819

org.apache.hadoop.hive.ql.parse.SemanticException:Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 03e679cf-420d-44f5-ae1c-79784c114819