Hi All, I was testing spark dynamic resource allocation in spark. By default I see "spark-thrift-sparkconf.conf" contains all the dynamic allocation properties. But when I run the spark job "spark-shell --master yarn --num-executors 5 --executor-memory 3G", I expect it complain as I've requested number of executor in the job itself. Then I modifed the custom spark-defaults.conf and added dynamic allocation properties: spark.dynamicAllocation.enabled true spark.dynamicAllocation.initialExecutors 1 spark.dynamicAllocation.maxExecutors 5 spark.dynamicAllocation.minExecutors 1 And when I run the same job, I see below messages : 16/05/23 09:18:54 WARN SparkContext: Dynamic Allocation and num executors both set, thus dynamic allocation disabled. Also print below messages if needed more resources. My doubt is is dynamic allocation is defined by default? Which config we should define dynamic allocation properties? 6/05/23 09:39:47 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 4) 16/05/23 09:39:48 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 5) ... View more

Hdfs can be accessed in R by specifying namenode host (hdfs://<hostname>:/user/test), but in case of namenode failover it won't work. So we should set hdfs config directory in R so it recognizes the hadoop namenode and failover configurations.. Set Hadoop and Spark home and config directories into R environment as below: # set up the SPARK_HOME Sys.setenv (SPARK_HOME="/usr/hdp/current/spark-client") #set up the HADOOP config dir Sys.setenv (YARN_CONF_DIR="/usr/hdp/current/hadoop-client/conf") Sys.setenv (HADOOP_CONF_DIR="/usr/hdp/current/hadoop-client/conf") # read the data file sc = sparkR.init(sparkPackages="com.databricks:spark-csv_2.11:1.0.3") sqlContext = sparkRSQL.init(sc) people = read.df(sqlContext, "hdfs://HDFS-HA/users/people.json", "json") head(people) ... View more