Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

[#1]

I am unable to configure spark as a execution engine in hive. Could you please help me with this

i am using HDP 2.4.

[#2 · Accepted solution]

Hello Kiran

Spark is not yet a GA feature in Hive, still very much in dev phase. You can however use SparkSQL to issue queries in a hive context to use Hive tables.

[#3]

Use hive.execution.engine=tez for better performance

[#4]

When can we expect spark to be GA with HIVE?

Thanks,

Vishal

[#5]

Can you please provide logs or error messages which you are getting?

Also did you tried using below parameters on hive shell?

set hive.execution.engine=spark;

set spark.master=yarn-client;

set spark.eventLog.enabled=true;

set spark.executor.memory=512m;

set spark.executor.cores=2;

[#6]

We support MR or Tez as the execution engine for Hive. We also support SparkSQL.

[#7]

Hi..

By setting the above parameters on hive shell,

it is giving correct output for simple select * from table name query,

but is giving error as

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

when we do select count(*) from table name query.

I have also added spark -assembly-hadoop jar in hive/lib

please guide me ,i am new to spark