How to get spark.executor.memory size of spark session?

[#1]

I am using Spark 2.3.2.3.1.0.0-78.

I tried to use:

spark_session.sparkContext._conf.get('spark.executor.memory')

but I only received 'None'.

Can someone help me, please?

[#2 · Accepted solution]

Hi @sonnh 

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

Launch the Spark Shell by passing the Executor Memory:

[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'

Launch the Spark Shell by without passing the Executor Memory:

[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

Reference:

https://spark.apache.org/docs/latest/configuration.html

[#3]

Tks @RangaReddy  My purpose is to collect a series of pagings from an RDBMS and compare it with JVM_HEAP_MEMORY. Do you find this approach acceptable? I believe it could help alleviate the issue of small files on HDFS.  I'm facing difficulties in calculating the size of the DataFrame. It seems there's no straightforward way to accomplish it

[#4]

The following articles will help to identify the small file issue:

1. https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-locat...

2. https://sauravagarwaldigital.medium.com/too-small-data-solving-small-files-issue-using-spark-b7ef668...

[#5]

To calculate the DataFrame size, you can use SizeEstimator class.