Performance issue with simple reproducible case
Hi,
This is a reproducible, simple issue where the performance is surprisingly bad. It is a follow-up to the case under this link, where initially a stoackoverflow issue occurred.
The script below ran for 26 hours in over 8 cores at full calculation as is seen in the hardware statistics.
Of course the object is "quite large"- but in similar operations with such an object it does not take so long. The generated size was 42.5GB in the 8 parquet files on HDFS.
Here is the code:
from pyspark.sql import SparkSession
from pyspark.sql.functions import rand
ss = SparkSession.builder.appName("test_replication") \
.config("spark.kryoserializer.buffer.max.mb", "2047") \
.config('spark.sql.execution.arrow.pyspark.enabled', "true") \
.config("spark.driver.maxResultSize", "16G") \
.config("spark.driver.memory", "4G") \
.config("spark.executor.memory", "16G") \
.config("spark.dynamicAllocation.maxExecutors","8") \
.config("spark.executor.instances", "2") \
.config("spark.executor.cores", "4") \
.config("spark.dynamicAllocation.enabled", "true") \
.config("spark.driver.extraJavaOptions", "-Xss1024m") \
.config("spark.executor.extraJavaOptions", "-Xss1024m") \
.config("spark.yarn.tags","dev") \
.getOrCreate()
rows=2350000
cols=2500
hdfs_dir="/destination/on/hdfs"
data = ss.range(rows)
for i in range(cols):
data=data.withColumn(f'col{i}', rand() * 2 -1)
data.write.format("parquet").mode("overwrite").save(f"{hdfs_dir}/test.parquet")
Am I doing something wrong?
Edit: I see in the log of the applicationHistory the following element that surprises me - is this normal?
"Storage Level":{"Use Disk":false,"Use Memory":false,"Deserialized":false,"Replication":1},
Edit2: Is this due to no cache() or persist() in place?

