Getting Warning: Maximum heap size rounded up to 512 MB while running spark-shell

[#1]

Even after passing the executor memory and driver memory as 1 Gb still it is getting maximum heap size rounded to 512 MB

spark-shell --conf spark.executor.memory=1g --conf spark.driver.memory=1g
Warning: Maximum heap size rounded up to 512 MB

Anything that is missing?

[#2 · Accepted solution]

I think you need to verify the yarn and spark resources are configured properly. If yes then go and check from spark ui, it will show driver memory and executor memory. It is coming as expected then safely you can ignore it.