Hi Brian, You shouldn't need to use exlode, that will create a new row for each value in the array. The reason max isn't working for your dataframe is because it is trying to find the max for that column for every row in you dataframe and not just the max in the array. Instead you will need to define a udf and call the udf within withColumn from pyspark.sql.functions import udf
def maxList(list):
max(list)
maxUdf==udf(scoreToCategory, FloatType())
df = df.withColumn('WF_Peak', maxUdf('wfdataseries')) As for using pandas and converting back to Spark DF, yes you will have a limitation on memory. toPandas calls collect on the dataframe and brings the entire dataset into memory on the driver, so you will be moving data across network and holding locally in memory, so this should only be called if the DF is small enough to store locally.
... View more
