01-23-2019 05:20 AM
My name is Sankar.
Part of Statiscal modelling team from our company.
Recently we are asked to pull data from hadoop from CDSW using R workbench thorugh spark job .
In that process, i am encountering some issues while connecting to Yarn cluster .
Below is the code used for the same.
conf <- spark_config()
conf$spark.submit.deployMode <- "cluster"
conf$spark.executor.instances <- 8
conf$spark.executor.cores <- 4
conf$spark.executor.memory <- "10G"
con<-spark_connect(master = "yarn",config = conf)
And this is the error i am getting.
01-23-2019 05:29 AM - edited 01-23-2019 05:29 AM
Thanks for posting your query with us!
From the error message which you have posted, it seems your CDSW session is not able to reach the YARN resource manager
Does your normal spark jobs from CDSW is running fine ?