My name is Sankar.
Part of Statiscal modelling team from our company.
Recently we are asked to pull data from hadoop from CDSW using R workbench thorugh spark job .
In that process, i am encountering some issues while connecting to Yarn cluster .
Below is the code used for the same.
conf <- spark_config()
conf$spark.submit.deployMode <- "cluster"
conf$spark.executor.instances <- 8
conf$spark.executor.cores <- 4
conf$spark.executor.memory <- "10G"
con<-spark_connect(master = "yarn",config = conf)
And this is the error i am getting.
Thanks for posting your query with us!
From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager
Does your normal spark jobs from CDSW is running fine ?
Yes i am able to run the normal spark jobs.and the same code works fine with execution at driver/client side.