Support Questions
Find answers, ask questions, and share your expertise

Specify Yarn Queues in RStudio

Specify Yarn Queues in RStudio

Contributor

How do I specify the yarn queue when writing a map reduce job in RStudio?

3 REPLIES 3

Re: Specify Yarn Queues in RStudio

@Nasheb Ismaily

Can you please try like this if you are using hivecontext with spark on yarn?,

Sys.setenv(SPARK_HOME="/usr/hdp/current/spark-client/").libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"),.libPaths()))
library(SparkR)
sc <- SparkR::sparkR.init(master = "yarn-client", sparkEnvir = list(spark.yarn.queue="queue-name")) 
hiveContext <- sparkRHive.init(sc)

Ref: https://community.hortonworks.com/questions/31713/sparkr-connect-to-hadoop-cluster.html

Re: Specify Yarn Queues in RStudio

@Nasheb Ismaily

please try this:

rhive.set("tez.queue.name","queue1")

Re: Specify Yarn Queues in RStudio

New Contributor

When starting spark R, the Spark Session is already generated.
You need to stop the current session and spin up a new one to set the desired settings.

I use the following

sparkR.stop()
sparkR.session(
    # master="local[2]",              # local master
    master="yarn",                    # cluster master
    appName="my_sparkR",
    sparkConfig=list(
        spark.driver.memory="4g",
        spark.executor.memory="2g",
        spark.yarn.queue="your_desired_queue"
    )
)


Verify from the Spark monitoring page that the settings updated correctly.