Yarn container limits für Spark jobs

[#1]

We are getting the following error when running a PySpark job on YARN:

"Container killed by YARN for exceeding memory limits. 12.3 GB of 12 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead."

My question is not so much why we get this error message but where does the value for the physical memory limit come from?
In our configuration we set:

spark.executor.memory=8g
spark.yarn.executor.memoryOverhead=2.5g

As far as I know, these are the two config values that influence the container size in YARN for a Spark job.
So I would expect the physical memory limit to be 10.5 GB and not 12 GB as stated in the error message.
Is there any further parameter that I'm not aware of?

Thanks for clearification!
Alex

[#2]

Hi @Alexander Schätzle ,

Can you share the details on how are you setting these two properties? What is your spark version?

Can you grep the line in your application log like "YarnAllocator":

INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 10692 MB memory (including 2500 MB of overhead)

That's what I got from Spark 1.6 (spark.yarn.executor.memoryOverhead=2500) and Spark 2.x (spark.executor.memoryOverhead=2500m).

I don't think we can even do "2.5g" right? I didn't try programmatically though.

In spark1.6.x:

- spark.executor.memoryOverhead=2.5g, would not be picked up and you'll end up with default 384Mb.

- spark.yarn.executor.memoryOverhead=2.5g, you would get an exception like 'Exception in thread "main" java.lang.NumberFormatException: For input string: "2.5g"'

In spark2.x:

- conf spark.yarn.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5"

- conf spark.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5"

[#3]

Hi @dbompart,

our Spark version is 2.2.0 (HDP 2.6.3).
I think it's not an issue how we set the parameter, because we actually set it this way in spark2-defaults:

spark.yarn.executor.memoryOverhead=2500

And here if to output from the log:

INFO YarnAllocator: Will request 12 executor container(s), each with 4 core(s) and 10692 MB memory (including 2500 MB of overhead)

So the values seem to be right and the request fits my expectations (8g executor memory + 2.5g overhead).

But I'm confused why the container is not killed when exceeding 10.5 GB but 12 GB:

 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 13.3 GB of 12 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

Is there any further parameter that I'm missing here?