I have a dataframe(df) created from hive table(warehouse directory). On the df I have performed group and count and have created a new dataframe( dfGrp). Now on each row of the dfGrp I wanted to call a method. How do I traverse every row of the dataframe? My code looks like below: val df = sqlContext.read.parquet("/user/hive/warehouse/xxx") val dfGrp = df.groupBy("col4").count().select(col("col4"),col("count").as("BadCount")) I wanted to call the method for each row of dfGrp. DataReport(String, String, Double, Double, Dataframe). This method return type is Dataframe.
case class DataReport(query:String, col4:String, BadCount:Double, df:DataFrame)
How I call the method for every row of the dataframe or RDD.
I can't understand exactly what you want to do with the DataReport case class... Anyway, if you want to perform a method on each row of a dataframe you have two options: