Hi @D Mortimer, I presume the problem is converting to RDD and processing as chunks, instead of that, I could think of by porting the custom function into Register UDF and apply the logic on the data(frame) which you have retrieved into t , and not to persist the data into memory, so that the UDF get applied across multiple executors while streaming the data and writes back to the table ( shuffles if needed). ... View more