Reply
Explorer
Posts: 7
Registered: ‎01-23-2019

Got an error while connecting to yarn cluster through spark job

Hello Team,

 

My name is Sankar.

 

Part of Statiscal modelling team from our company.

 

Recently we are asked to pull data from hadoop from CDSW using R workbench thorugh spark job .

 

In that process, i am encountering some issues while connecting to Yarn cluster .

 

Below is the code used for the same.

 

conf <- spark_config()
conf$spark.submit.deployMode <- "cluster"
conf$spark.executor.instances <- 8
conf$spark.executor.cores <- 4
conf$spark.executor.memory <- "10G"
con<-spark_connect(master = "yarn",config = conf)

 

And this is the error i am getting.

 

"Error in spark_yarn_cluster_get_resource_manager_webapp() :
  Failed to find online resource manager under High Availability cluster."
 
 
Kindly help in this one.
 
Thanks and Regards
Sankar Narayana
Highlighted
Posts: 57
Topics: 0
Kudos: 8
Solutions: 5
Registered: ‎05-15-2018

Re: Got an error while connecting to yarn cluster through spark job

[ Edited ]

Hello @Sankar

 

Thanks for posting your query with us!

 

From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager

 

Does your normal spark jobs from CDSW is running fine ?

Thanks,
Satz
Explorer
Posts: 7
Registered: ‎01-23-2019

Re: Got an error while connecting to yarn cluster through spark job

Yes i am able to run the normal spark jobs.and the same code works fine with execution at driver/client side.