question Re: Dynamic allocation on Spark Standalone cluster in Support Questions

Dynamic allocation on Spark Standalone cluster

jphan — Tue, 04 Oct 2016 16:35:14 GMT

I used Spark Standalone cluster (Ver 1.6.1).

By the way, I want to dynamic allocation setting.

In this case, I have Question.

All dynamic allocation parameter that do operate in standalone mode?

Such as, 'spark.dynamicAllocation.maxExecutors' or 'spark.dynamicAllocation.minExecutors'.

And the executor parameters does it work properly?

Such as, 'spark.executor.cores' and 'spark.executor.memory'.

I look forward to response.

Thanks,

lgeorge — Tue, 04 Oct 2016 23:14:54 GMT

@Han Jeongphill, while Spark standalone is not officially supported by Hortonworks at this time, dynamic resource allocation is available for standalone mode at the community level. For prerequisites, configuration info, and how to start dynamic allocation on a standalone cluster, see https://spark.apache.org/docs/1.6.1/job-scheduling.html#dynamic-resource-allocation.

jphan — Wed, 05 Oct 2016 12:35:57 GMT

@lgeorge, The link you posted has been a great help to me.

Thanks.