Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn container limits für Spark jobs

Yarn container limits für Spark jobs

New Contributor

We are getting the following error when running a PySpark job on YARN:

"Container killed by YARN for exceeding memory limits. 12.3 GB of 12 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead."

My question is not so much why we get this error message but where does the value for the physical memory limit come from?
In our configuration we set:

spark.executor.memory=8g
spark.yarn.executor.memoryOverhead=2.5g

As far as I know, these are the two config values that influence the container size in YARN for a Spark job.
So I would expect the physical memory limit to be 10.5 GB and not 12 GB as stated in the error message.
Is there any further parameter that I'm not aware of?

Thanks for clearification!
Alex

2 REPLIES 2

Re: Yarn container limits für Spark jobs

Contributor

Hi @Alexander Schätzle ,

Can you share the details on how are you setting these two properties? What is your spark version?

Can you grep the line in your application log like "YarnAllocator":

INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 10692 MB memory (including 2500 MB of overhead)

That's what I got from Spark 1.6 (spark.yarn.executor.memoryOverhead=2500) and Spark 2.x (spark.executor.memoryOverhead=2500m).

I don't think we can even do "2.5g" right? I didn't try programmatically though.

In spark1.6.x:

- spark.executor.memoryOverhead=2.5g, would not be picked up and you'll end up with default 384Mb.

- spark.yarn.executor.memoryOverhead=2.5g, you would get an exception like 'Exception in thread "main" java.lang.NumberFormatException: For input string: "2.5g"'

In spark2.x:

- conf spark.yarn.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5"

- conf spark.executor.memoryOverhead=2.5g (deprecated warning), you would get an exception like "Exception in thread "main" java.lang.NumberFormatException: Size must be specified as bytes (b), kibibytes (k), mebibytes (m), gibibytes (g), tebibytes (t), or pebibytes(p). E.g. 50b, 100k, or 250m. Fractional values are not supported. Input was: 2.5"

Highlighted

Re: Yarn container limits für Spark jobs

New Contributor

Hi @dbompart,

our Spark version is 2.2.0 (HDP 2.6.3).
I think it's not an issue how we set the parameter, because we actually set it this way in spark2-defaults:

spark.yarn.executor.memoryOverhead=2500

And here if to output from the log:

INFO YarnAllocator: Will request 12 executor container(s), each with 4 core(s) and 10692 MB memory (including 2500 MB of overhead)

So the values seem to be right and the request fits my expectations (8g executor memory + 2.5g overhead).

But I'm confused why the container is not killed when exceeding 10.5 GB but 12 GB:

 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 13.3 GB of 12 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

Is there any further parameter that I'm missing here?

Don't have an account?
Coming from Hortonworks? Activate your account here