Make sure dfprocessed datafrmae doesn't contains any empty rows.  In Spark, you can identify and filter out empty rows in a DataFrame using the filter operation. Empty rows typically have null or empty values across all columns.   // Identify and filter out empty rows val nonEmptyRowsDF = df.filter(not(df.columns.map(col(_).isNull).reduce(_ || _)))   This code uses the filter operation along with the not function and a condition that checks if any column in a row is null. It then removes rows where all columns are null or empty. If you want to check for emptiness based on specific columns, you can specify those columns in the condition:   val columnsToCheck = Array("column1", "column2", "column3") val nonEmptyRowsDF = df.filter(not(columnsToCheck.map(col(_).isNull).reduce(_ || _)))   Adjust the column names based on your DataFrame structure. The resulting nonEmptyRowsDF will contain rows that do not have null or empty values in the specified columns. ... View more