question Re: spark thrift server not started in Support Questions

spark thrift server not started

mike_bronson7 — Tue, 06 Feb 2018 02:55:29 GMT

we have ambari cluster with 3 worker machines ( each worker have 8G memory )

when we start the spark thrift server on master01/03 machines we get the following errors

Warning: Master yarn-client is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
18/02/05 18:12:52 WARN Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs.
18/02/05 18:12:53 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (10240+1024 MB) is above the max threshold (6144 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocat
ion-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

please advice - what these errors means ?

about - Required executor memory (10240+1024 MB) , what this memory values and how to set the parameters in spark or other in order to solve these isshue ?

Re: spark thrift server not started

dhyun — Tue, 06 Feb 2018 03:08:41 GMT

Hi, @Michael Bronson

`spark.executor.memory` seems to be 10240.

Please change it in your Ambari, `spark-thrift-conf`.

Re: spark thrift server not started

mike_bronson7 — Tue, 06 Feb 2018 03:15:37 GMT

thank you , could you please explain about this variable and who is responsible to set it ? ( I mean this value set by the cluster itself? ) , on the first thinking it seems that workers not have the right memory so I increase the workers memory to 32G insted 8G

Re: spark thrift server not started

dhyun — Wed, 07 Feb 2018 01:22:33 GMT

It's a memory size for Spark executor (worker). And, there is additional overhead in Spark executor. You need to set a proper value by yourself. Of course, in YARN environment, the memory (+ overhead) should be smaller than the limitation of YARN container. So, Spark shows you the error message.

It's an application property. For normal Spark jobs, users are responsible because each app can set their `spark.executor.memory` with `spark-submit`. For Spark Thrift Server, admins should manage that properly when they adjust YARN configuration.

For more information, please see this. http://spark.apache.org/docs/latest/configuration.html#application-properties