Skip to main content
KPG1
KPG1
Gold ⭐️
January 24, 2023
Solved

Getting Warning: Maximum heap size rounded up to 512 MB while running spark-shell

Even after passing the executor memory and driver memory as 1 Gb still it is getting maximum heap size rounded to 512 MB

 

spark-shell --conf spark.executor.memory=1g --conf spark.driver.memory=1g
Warning: Maximum heap size rounded up to 512 MB

 

Anything that is missing?

RangaReddy
RangaReddyAnswer
Gold ⭐️
October 25, 2023

I think you need to verify the yarn and spark resources are configured properly. If yes then go and check from spark ui, it will show driver memory and executor memory. It is coming as expected then safely you can ignore it.

View in thread

1 reply

RangaReddy
Gold ⭐️
RangaReddyAnswer
Gold ⭐️
October 25, 2023

I think you need to verify the yarn and spark resources are configured properly. If yes then go and check from spark ui, it will show driver memory and executor memory. It is coming as expected then safely you can ignore it.

Translate
Terms & ConditionsAccessibility statement