@yvora Thanks for the response. 1. Can you try setting spark.yarn.stagingDir to hdfs:///user/tmp/ ? This is not working . 2. Can you please share which spark config are you trying to set which require RM address? I am trying to run the Spark application through java program , so when the master is yarn , by default it connects to resource manager @ 0.0.0.0:8032 in order to override this property , I need to set the same in spark configuration i.e sparkConf.set("spark.hadoop.yarn.resourcemanager.hostname",resourcemanagerHostname); sparkConf.set("spark.hadoop.yarn.resourcemanager.address",resourcemanagerAddress); But the problem is when the I have HA enabled the resource manager How to I connect to it . And some idea I got about my question is : There is way to achieve this in the Spark context as below . JavaSparkContext jsc = new JavaSparkContext(sparkConf); jsc.hadoopConfiguration().addResource(new Path(hadoopClusterSiteFilesBasePath + "core-site.xml")); jsc.hadoopConfiguration().addResource(new Path(hadoopClusterSiteFilesBasePath + "hdfs-site.xml")); jsc.hadoopConfiguration().addResource(new Path(hadoopClusterSiteFilesBasePath + "mapred-site.xml")); jsc.hadoopConfiguration().addResource(new Path(hadoopClusterSiteFilesBasePath + "yarn-site.xml")); But it needs the resource manager and staging directory configuration even before creating the context ,so there is problem . And what I am looking for is something like above for the SparkConguration class/object . Thanks , Param. ... View more

@amankumbare Thanks for responding, Sumit and me work in the same team . It is happening for all topics and jdk we using is 7 . The main issue here is the content of the znode of the broker has no host and port information updated . ["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1} -Is it the expected behavior ? -And it happens when listener value is PLAINTEXTSASL://xxxx.domain.com:9092 and everything is fine when it is just PLAINTEXT . Thanks in advance , Param. ... View more

@Sandeep Nemuri Thanks its worked ..I tried this already but forgot to create the file on each node ,now its fine. And I just got one more question here : If I run in spark app YARN mode I can set the memory parameter through sparkconfiguration using spark.yarn.driver.memoryOverhead properties , Is something similar available for the standalone and local mode ? Thanks in advance , Param. ... View more

Sorry for the delayed reply ...I got busy in some work. @Artem Ervits thanks a lot for all the responses . I was able to achieve this by setting the spark configuration as below ;- sparkConfig.set("spark.hadoop.yarn.resourcemanager.hostname","XXXXX"); sparkConfig.set("spark.hadoop.yarn.resourcemanager.address","XXXXX:8032"); sparkConfig.set("spark.yarn.access.namenodes", "hdfs://XXXXX:8020,hdfs://XXXX:8020"); sparkConfig.set("spark.yarn.stagingDir", "hdfs://XXXXX:8020/user/hduser/"); sparkConfig.set("--deploy-mode", deployMode); Thanks , Param. ... View more

@Artem Ervits , Thanks again ! And Sorry If I am asking too many questions here . What actually I am looking for is ..I should not use the spark-submit script as per the project requirement , So the cluster configuration I am passing through the spark config as given below . SparkConf sparkConfig = new SparkConf().setAppName("Example App of Spark on Yarn"); sparkConfig.set("spark.hadoop.yarn.resourcemanager.hostname","XXXX"); sparkConfig.set("spark.hadoop.yarn.resourcemanager.address","XXXXX:8032"); And it is able to identify the Resource Manager but it failing because it is not identifying the file system . Though I am setting the hdfs file system configuration as well. sparkConfig.set("fs.defaultFS", "hdfs://xxxhacluster"); sparkConfig.set("ha.zookeeper.quorum", "xxx:2181,xxxx:2181,xxxx:2181"); And it assuming it as the local file system. And error I am getting in the Resource Manager is exited with exitCode: -1000 due to: File file:/tmp/spark-0e6626c2-d344-4cae-897f-934e3eb01d8f/__spark_libs__1448521825653017037.zip does not exist Thanks and Regards, Param. ... View more