question Spark submit multiple configurations in Archives of Support Questions (Read Only)

Spark submit multiple configurations

Former Member — Fri, 26 May 2017 21:34:25 GMT

The docs here same to place it in key value format https://spark.apache.org/docs/1.6.1/running-on-yarn.html

But unfortunately did not give a clear example I want to have 2 configurations set.

spark-submit --conf "spark.hadoop.parquet.enable.summary-metadata=false;spark.yarn.maxAppAttempts=1" etc..

Is this the correct way of doing it and if not what would be the correct way.

I am using spark 1.6

Thank you

Re: Spark submit multiple configurations

gbrahmi — Fri, 26 May 2017 23:45:57 GMT

@elliot gimple

the correct way to pass multiple configuration options is to specify them individually. The following should work for your example:

spark-submit --conf spark.hadoop.parquet.enable.summary-metadata=false --conf spark.yarn.maxAppAttempts=1

As always if you like the answer please up vote the answer.

Re: Spark submit multiple configurations

Former Member — Fri, 26 May 2017 23:50:53 GMT

I had one more question if I need the arguments to be in quotes then --conf "A" --conf "B" for the arguments doesnt work. Just curious if you happen to know how pass two arguments in quotes for the spark submit.

Thanks for the answer :).

Re: Spark submit multiple configurations

gbrahmi — Fri, 26 May 2017 23:56:47 GMT

I believe single quote should work. Try --conf 'some.config' --conf 'other.config'.

Re: Spark submit multiple configurations

AKR — Tue, 10 Dec 2019 07:32:53 GMT

Hi,

The correct way to pass the multiple configurations is that it should be passed along with the --conf.

Ex:

spark-submit --conf org.spark.metadata =false --conf spark.driver.memory=10gb

Thanks