question Re: How to get spark.executor.memory size of spark session? in Support Questions

How to get spark.executor.memory size of spark session?

sonnh — Mon, 14 Aug 2023 08:34:57 GMT

I am using Spark 2.3.2.3.1.0.0-78.

I tried to use:

spark_session.sparkContext._conf.get('spark.executor.memory')

but I only received 'None'.

Can someone help me, please?

Re: How to get spark.executor.memory size of spark session?

RangaReddy — Thu, 17 Aug 2023 05:29:58 GMT

Hi @sonnh

If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.

Launch the Spark Shell by passing the Executor Memory:

[root@local ~]# pyspark --conf spark.executor.memory=1g >>> spark.conf.get("spark.executor.memory") u'1g'

Launch the Spark Shell by without passing the Executor Memory:

[root@local ~]# pyspark >>> spark.conf.get("spark.executor.memory") py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. : java.util.NoSuchElementException: spark.executor.memory ...

Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.

Reference:

https://spark.apache.org/docs/latest/configuration.html

Re: How to get spark.executor.memory size of spark session?

sonnh — Fri, 18 Aug 2023 07:24:46 GMT

Tks @RangaReddy My purpose is to collect a series of pagings from an RDBMS and compare it with JVM_HEAP_MEMORY. Do you find this approach acceptable? I believe it could help alleviate the issue of small files on HDFS. I'm facing difficulties in calculating the size of the DataFrame. It seems there's no straightforward way to accomplish it

Re: How to get spark.executor.memory size of spark session?

RangaReddy — Sun, 20 Aug 2023 15:54:16 GMT

The following articles will help to identify the small file issue:

1. https://community.cloudera.com/t5/Community-Articles/Identify-where-most-of-the-small-file-are-located-in-a-large/ta-p/247253

2. https://sauravagarwaldigital.medium.com/too-small-data-solving-small-files-issue-using-spark-b7ef66827a24

Re: How to get spark.executor.memory size of spark session?

RangaReddy — Sun, 20 Aug 2023 15:54:49 GMT

To calculate the DataFrame size, you can use SizeEstimator class.