Created 01-18-2018 03:53 AM
Team,
Iam getting below error while initiating a connecting to HIVE through R studio using spark2
Code:
Sys.setenv("SPARK_MAJOR_VERSION"="2") ycc <- rxSparkConnect(consoleOutput=TRUE,executorMem="4g",driverMem="4g", executorOverheadMem="4g",executorCores=1,numExecutors=5, persistentRun=TRUE,extraSparkConfig='--conf spark.yarn.queue=default')
Error:
No Spark applications can be reused. It may take 1 to 2 minutes to launch a new Spark application. Error in rxRemoteExecute(computeContext, shellCmd, schedulerJobInstance) : Error while running RevoSparkSubmitLauncher ERROR: Failed to start a Spark application, perhaps because the cluster is busy with other jobs. Please check YARN usage and try again. rxSetComputeContext(ycc) Error in is(computeContext, "RxSpark") : object 'ycc' not found
I see their is a enough resources avaliable on the cluter and for the particular queue
Created 01-18-2018 06:12 PM
Try to export "Spark_Home” to the search path for R packages so that we can use SparkR, and initialize a SparkR session.
Sys.setenv("SPARK_MAJOR_VERSION"="2") Sys.setenv("SPARK_HOME"="/usr/hdp/2.6.1.0-129/spark2") ycc<-rxSparkConnect(consoleOutput=TRUE,executorMem="10g",driverMem="10g",executorOverheadMem="10g",executorCores=1,numExecutors=5,extraSprakConfig="--conf spark.yarn.queue=default")
More info on link: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-r-scaler-sparkr
Hope this helps you.
Created 01-19-2018 02:20 AM
Hi Sridhar, Thanks for your response.. I tried with pointing to spark2 conf dir but got the same error
Created 01-25-2018 04:31 PM
May I know what's the error message you are getting?
Created 01-19-2018 02:17 AM
Hi Sridhar, thanks for your reply . tried with the option... getting same error..