Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

what are the different options available to pass properites to Spark Application

what are the different options available to pass properites to Spark Application

New Contributor

I have an spark application which writes output on HDFS directory. HDFS directory differs for each environment and the list of properties will grow over the period of time. What are the different ways to pass and access custom properties with in an spark application ?

4 REPLIES 4

Re: what are the different options available to pass properites to Spark Application

Super Guru

If your application is in java why not simply pass in the parameter via program args

public static void main(String[] args){

and then simply pull your args during run time.

Re: what are the different options available to pass properites to Spark Application

Super Guru

Got it. Another option is to use a properties file and load into a java property object. Here you parse for parameters you need. It is essentially limitless.

Re: what are the different options available to pass properites to Spark Application

New Contributor

I thought of that option but the number properties will increase overtime and it will be difficult to manager that.

Re: what are the different options available to pass properites to Spark Application

Expert Contributor

You can manage all of the parameters in a file and pass them to the spark-submit via --properties-files

You can retrieve these properties by typing:

sc.getConf.get("name.of.property")