I couldn't come up with anything better than manually scanning the DataFrame to check if all values in a column are NULL. Something like: // Returns the names of all empty columns of DataFrame
def getEmptyColNames(df: DataFrame): Seq[String] = {
df.cache()
val colNames: Seq[String] = df.columns
colNames.filter { (colName: String) =>
df.filter(df(colName).isNotNull).count() == 0
}
}
// Drops all empty columns of DataFrame
def dropEmptyCols(df: DataFrame): DataFrame = {
val emptyColNames: Seq[String] = getEmptyColNames(df)
if (emptyColNames.isEmpty) df
else df.drop(emptyColNames: _*)
}
val dfOriginal: DataFrame
val dfNonEmptyCols: DataFrame = dropEmptyCols(dfOriginal) @Junfeng Chen Were you able to find a more efficient / smarter way?
... View more
