Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Program executing Tez engine instead of Spark

avatar
Contributor

Hello All,

 

We have a Spark program which executes multiple queries and the tables are Hive tables. 

 

Currently the queries are executed using Tez engine from Spark.

 

I set it sqlContext.sql("SET hive.execution.engine=spark") in the program and understand that the queries/program would run as Spark. We are using HDP 2.6.5 version and Spark 2.3.0 version in our cluster. 

Can someone suggest that it is the correct way as we do not need to run the queries using Tez engine and Spark should run as it is .

 

In the config file /etc/spark2/conf/hive-site.xml, we do not have any specific engine property setup and we do have only kerberos, metastore property details.

 

Thanks

1 REPLY 1

avatar
Master Collaborator

Hi @adhishankarit 

Spark applications will run using spark engine and not a tez engine unlike hive. You no need to set any engine from spark side.

If you want to run hive queries then you can set engines like Tez, Spark, MR