Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to get spark.executor.memory size of spark session?

avatar
Contributor
I am using Spark 2.3.2.3.1.0.0-78.
I tried to use:
spark_session.sparkContext._conf.get('spark.executor.memory')
but I only received 'None'.

Can someone help me, please?

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Hi @sonnh 

 

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

 

Launch the Spark Shell by passing the Executor Memory:

 

 

 

[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'

 

 
Launch the Spark Shell by without passing the Executor Memory:
 

 

[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...

 

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

 

Reference:

 

https://spark.apache.org/docs/latest/configuration.html

View solution in original post

4 REPLIES 4

avatar
Master Collaborator

Hi @sonnh 

 

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

 

Launch the Spark Shell by passing the Executor Memory:

 

 

 

[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'

 

 
Launch the Spark Shell by without passing the Executor Memory:
 

 

[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...

 

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

 

Reference:

 

https://spark.apache.org/docs/latest/configuration.html

avatar
Contributor

Tks @RangaReddy  My purpose is to collect a series of pagings from an RDBMS and compare it with JVM_HEAP_MEMORY. Do you find this approach acceptable? I believe it could help alleviate the issue of small files on HDFS.  I'm facing difficulties in calculating the size of the DataFrame. It seems there's no straightforward way to accomplish it

avatar
Master Collaborator

avatar
Master Collaborator

To calculate the DataFrame size, you can use SizeEstimator class.