Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error "Failed to create Spark client for Spark session" with spark execution engine

avatar
Contributor

 

 

Below are the version details
CDP 7.1.8
CM : 7.8.1
HIVE : 3.1.3
Spark 2 Version: 2.4.8
Spark 3 Version: 3.3.0

ISSUE DECRIPTION :
We have table with more than 3 million rows. We are not able to execute conditional QUERY with "WHERE", COUNT, with Spark execution engine in hive
When we set to hive execution engine to spark (set hive.execution.engine=spark) we get the error mentioned below :

QUERY FAILED : SELECT * FROM test_.JOBS__PROJECT WHERE state = 'DONE' LIMIT 10;

ERROR : FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session c-47f2-aceb-22390502b303
Error: Error while compiling statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session d6d96da5-f2bc-47f2-aceb-22390502b303 (state=42000,code=30041)

We are able to execute the same query with execution engine set to tez and also able to execute from spark-shell.

Also to note we are successfully able to execute non-conditional query with Spark execution engine
SUCCESS QUERY : SELECT * FROM test_.JOBS__PROJECT LIMIT 10;

1 REPLY 1

avatar
Master Collaborator

Hi @dmharshit 

 

With the below log, difficult to provide a solution because it don't have full logs. Please create a cloudera case to check the logs and provide a solution.

 

ERROR : FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session c-47f2-aceb-22390502b303
Error: Error while compiling statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session d6d96da5-f2bc-47f2-aceb-22390502b303 (state=42000,code=30041)