Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark submit multiple configurations

Solved Go to solution

Spark submit multiple configurations

The docs here same to place it in key value format https://spark.apache.org/docs/1.6.1/running-on-yarn.html

But unfortunately did not give a clear example I want to have 2 configurations set.

spark-submit --conf "spark.hadoop.parquet.enable.summary-metadata=false;spark.yarn.maxAppAttempts=1" etc..

Is this the correct way of doing it and if not what would be the correct way.

I am using spark 1.6

Thank you

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Spark submit multiple configurations

Explorer
@elliot gimple

the correct way to pass multiple configuration options is to specify them individually. The following should work for your example:

spark-submit --conf spark.hadoop.parquet.enable.summary-metadata=false --conf spark.yarn.maxAppAttempts=1

As always if you like the answer please up vote the answer.

View solution in original post

4 REPLIES 4
Highlighted

Re: Spark submit multiple configurations

Explorer
@elliot gimple

the correct way to pass multiple configuration options is to specify them individually. The following should work for your example:

spark-submit --conf spark.hadoop.parquet.enable.summary-metadata=false --conf spark.yarn.maxAppAttempts=1

As always if you like the answer please up vote the answer.

View solution in original post

Highlighted

Re: Spark submit multiple configurations

I had one more question if I need the arguments to be in quotes then --conf "A" --conf "B" for the arguments doesnt work. Just curious if you happen to know how pass two arguments in quotes for the spark submit.

Thanks for the answer :).

Highlighted

Re: Spark submit multiple configurations

Explorer

I believe single quote should work. Try --conf 'some.config' --conf 'other.config'.

Highlighted

Re: Spark submit multiple configurations

Cloudera Employee

Hi,

 

The correct way to pass the multiple configurations is that it should be passed along with the --conf.

Ex:

spark-submit --conf org.spark.metadata =false --conf spark.driver.memory=10gb

 

Thanks

AK

Don't have an account?
Coming from Hortonworks? Activate your account here