Can someone help me, please?
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to get spark.executor.memory size of spark session?
- Labels:
-
Apache Spark
Created ‎08-14-2023 01:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created on ‎08-16-2023 10:27 PM - edited ‎08-16-2023 10:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @sonnh
If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.
Launch the Spark Shell by passing the Executor Memory:
[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'
[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...
Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.
Reference:
Created on ‎08-16-2023 10:27 PM - edited ‎08-16-2023 10:29 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @sonnh
If you do not specify spark.executor.memory when using spark-submit or spark-shell, or pyspark, the default value for spark.executor.memory will be set to 1g.
Launch the Spark Shell by passing the Executor Memory:
[root@local ~]# pyspark --conf spark.executor.memory=1g
>>> spark.conf.get("spark.executor.memory")
u'1g'
[root@local ~]# pyspark
>>> spark.conf.get("spark.executor.memory")
py4j.protocol.Py4JJavaError: An error occurred while calling o66.get. :
java.util.NoSuchElementException: spark.executor.memory ...
Note: Spark is not able to detect/load the spark.executor.memory parameter value from launched spark shell.
Reference:
Created on ‎08-18-2023 12:23 AM - edited ‎08-18-2023 12:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Tks @RangaReddy My purpose is to collect a series of pagings from an RDBMS and compare it with JVM_HEAP_MEMORY. Do you find this approach acceptable? I believe it could help alleviate the issue of small files on HDFS. I'm facing difficulties in calculating the size of the DataFrame. It seems there's no straightforward way to accomplish it
Created ‎08-20-2023 08:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The following articles will help to identify the small file issue:
Created ‎08-20-2023 08:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
To calculate the DataFrame size, you can use SizeEstimator class.
