question Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell? in Archives of Support Questions (Read Only)

Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

kiranbhakre1 — Mon, 11 Apr 2016 18:09:00 GMT

I am unable to configure spark as a execution engine in hive. Could you please help me with this

i am using HDP 2.4.

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

nmaillard1 — Mon, 11 Apr 2016 18:12:58 GMT

Hello Kiran

Spark is not yet a GA feature in Hive, still very much in dev phase. You can however use SparkSQL to issue queries in a hive context to use Hive tables.

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

jyadav — Mon, 11 Apr 2016 18:20:05 GMT

Can you please provide logs or error messages which you are getting?

Also did you tried using below parameters on hive shell?

set hive.execution.engine=spark;

set spark.master=yarn-client;

set spark.eventLog.enabled=true;

set spark.executor.memory=512m;

set spark.executor.cores=2;

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

thejas — Tue, 12 Apr 2016 01:08:10 GMT

Use hive.execution.engine=tez for better performance

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

vshukla — Tue, 12 Apr 2016 01:14:26 GMT

We support MR or Tez as the execution engine for Hive. We also support SparkSQL.

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

vs255018 — Tue, 26 Jul 2016 17:16:35 GMT

When can we expect spark to be GA with HIVE?

Thanks,

Vishal

Re: Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

zssspagobi — Fri, 30 Sep 2016 20:47:28 GMT

Hi..

By setting the above parameters on hive shell,

it is giving correct output for simple select * from table name query,

but is giving error as

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

when we do select count(*) from table name query.

I have also added spark -assembly-hadoop jar in hive/lib

please guide me ,i am new to spark