Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Got an error while connecting to yarn cluster through spark job

Got an error while connecting to yarn cluster through spark job

Explorer

Hello Team,

 

My name is Sankar.

 

Part of Statiscal modelling team from our company.

 

Recently we are asked to pull data from hadoop from CDSW using R workbench thorugh spark job .

 

In that process, i am encountering some issues while connecting to Yarn cluster .

 

Below is the code used for the same.

 

conf <- spark_config()
conf$spark.submit.deployMode <- "cluster"
conf$spark.executor.instances <- 8
conf$spark.executor.cores <- 4
conf$spark.executor.memory <- "10G"
con<-spark_connect(master = "yarn",config = conf)

 

And this is the error i am getting.

 

"Error in spark_yarn_cluster_get_resource_manager_webapp() :
  Failed to find online resource manager under High Availability cluster."
 
 
Kindly help in this one.
 
Thanks and Regards
Sankar Narayana
2 REPLIES 2

Re: Got an error while connecting to yarn cluster through spark job

Rising Star

Hello @Sankar

 

Thanks for posting your query with us!

 

From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager

 

Does your normal spark jobs from CDSW is running fine ?

Thanks,
Satz

Re: Got an error while connecting to yarn cluster through spark job

Explorer

Yes i am able to run the normal spark jobs.and the same code works fine with execution at driver/client side.