How to get spark.executor.memory size of spark session?

4 replies
10655 views
Translate All

I am using Spark 2.3.2.3.1.0.0-78.

I tried to use:

spark_session.sparkContext._conf.get('spark.executor.memory')

but I only received 'None'.

Can someone help me, please?

Apache Spark

RangaReddyAnswer

Gold ⭐️

Hi @sonnh

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

Launch the Spark Shell by passing the Executor Memory:

[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'

Launch the Spark Shell by without passing the Executor Memory:

[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

Reference:

https://spark.apache.org/docs/latest/configuration.html

RangaReddyAnswer

Gold ⭐️

Hi @sonnh

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

Launch the Spark Shell by passing the Executor Memory:

[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'

Launch the Spark Shell by without passing the Executor Memory:

[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

Reference:

https://spark.apache.org/docs/latest/configuration.html

Translate

sonnhAuthor

Silver ⭐️⭐️⭐️

Tks @RangaReddy My purpose is to collect a series of pagings from an RDBMS and compare it with JVM_HEAP_MEMORY. Do you find this approach acceptable? I believe it could help alleviate the issue of small files on HDFS. I'm facing difficulties in calculating the size of the DataFrame. It seems there's no straightforward way to accomplish it

Translate

RangaReddy

Gold ⭐️

The following articles will help to identify the small file issue:

1. https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-located-in-a-large/ta-p/247253

2. https://sauravagarwaldigital.medium.com/too-small-data-solving-small-files-issue-using-spark-b7ef66827a24

Translate

RangaReddy

Gold ⭐️

To calculate the DataFrame size, you can use SizeEstimator class.

Translate