Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive on Spark CDH 5.7 - Failed to create spark client

avatar
Explorer

Hi All,

 

We are getting the error while executing the hive queries with spark engine.

 

Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask

 

The following properties are set to use spark as the execution engine instead of mapreduce:

set hive.execution.engine=spark;

set spark.executor.memory=2g;

 

I tried changing the following properties  also.

set yarn.scheduler.maximum-allocation-mb=2048;

set yarn.nodemanager.resource.memory-mb=2048;

set spark.executor.cores=4

set spark.executor.memory=4g;

set spark.yarn.executor.memoryOverhead=750

set hive.spark.client.server.connect.timeout=900000ms;

 

1 ACCEPTED SOLUTION

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Super Guru
You will need to check both HS2 log and Spark application log to get the real error message.

"Failed to create spark client" is too generic and it can be anything.

avatar
Explorer
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Explorer
may i ask do you setup the params just in your beeline sql script? do you need to change the configuration xml for hs2?

avatar
Explorer

Yes. Just in the hql file. Not anything in XML file