Support Questions

Find answers, ask questions, and share your expertise
Announcements
Welcome to the upgraded Community! Read this blog to see What’s New!

Unable to use hive.execution.engine=spark In HDP 2.4 / how to configure spark to use in hive shell?

avatar

I am unable to configure spark as a execution engine in hive. Could you please help me with this

i am using HDP 2.4.

1 ACCEPTED SOLUTION

avatar

Hello Kiran

Spark is not yet a GA feature in Hive, still very much in dev phase. You can however use SparkSQL to issue queries in a hive context to use Hive tables.

View solution in original post

6 REPLIES 6

avatar

Hello Kiran

Spark is not yet a GA feature in Hive, still very much in dev phase. You can however use SparkSQL to issue queries in a hive context to use Hive tables.

avatar
Cloudera Employee

Use hive.execution.engine=tez for better performance

avatar
New Contributor

When can we expect spark to be GA with HIVE?

Thanks,

Vishal

avatar

Can you please provide logs or error messages which you are getting?

Also did you tried using below parameters on hive shell?

set hive.execution.engine=spark;

set spark.master=yarn-client;

set spark.eventLog.enabled=true;

set spark.executor.memory=512m;

set spark.executor.cores=2;

avatar

We support MR or Tez as the execution engine for Hive. We also support SparkSQL.

avatar
New Contributor

Hi..

By setting the above parameters on hive shell,

it is giving correct output for simple select * from table name query,

but is giving error as

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

when we do select count(*) from table name query.

I have also added spark -assembly-hadoop jar in hive/lib

please guide me ,i am new to spark

Labels