Ah, sorry:) Yes, here you can't specify driver related parameters using <spark-opts>--driver-memory 10g</spark-opts> because your driver (oozie launcher job) is already launched before that point. It's a oozie launcher (which is a mapreduce job) launches your actual spark job and so spark-opts is not relevant. But the Oozie spark action doc says:
The configuration element, if present, contains configuration properties that are passed to the Spark job. This is shouldn't be spark configuration. It should be mapreduce configuration for launcher job.
Thank you so much for such a clear answer. I tried the steps mentioned below and set the individual values as 4096 and 3072 but my job failed due to "MAP capability required is more than the supported max container capability in the cluster". I checked the properties "mapreduce.map.memory.mb" and "mapreduce.map.java.opts" in mapred-site.xml and their values mentioned are 250 and -Xmx200m. So This might be the reason my job is getting killed as it is requesting container size more than default values.
Any workaround for this? If i update the values in mapred-site.xml to above mentioned values, then which services i need to restart to reflect those changes? Or can it be resolved in any other way? By the way I am running HDP 2.5
Just a small question. My Spark oozie workflows keeps on running from long time. When i checked oozie logs i found it is trying to connect to port 8032 on sandbox.hortonworks.com. I donot know why it is going to 8032 instead of 8050 although i have mentioned 8050 in my job.properties.
@rahul gulati Earlier I observed that this similar exception occurred at the time of launching of Oozie workflow. Can you try to set following memory related parameter in Oozie workflow.xml with some higher value like 1024mb so that workflow launches successfully.