question How to get the non group by columns in spark structured streaming in Support Questions

question How to get the non group by columns in spark structured streaming in Support Questions https://community.cloudera.com/t5/Support-Questions/How-to-get-the-non-group-by-columns-in-spark-structured/m-p/200511#M162532 Hi, Below is the input schema and output schema.i/p: row_id,ODS_WII_VERB,stg_load_ts,other_columns o/p: get the max timestamp group by row_id and ODS_WII_VERBissue: As we use only row_id and ODS_WII_VERB in the group by clause we are unable to get the other columns. How to get other columns as well. We tried creating a spark sql subquery but it seems spark sub query is not working in spark structured streaming. How to resolve this issue. code snippet val csvDF = sparkSession .readStream .option("sep", ",") .schema(userSchema) .csv("C:\\Users\\M1037319\\Desktop\\data") val updatedDf = csvDF.withColumn("ODS_WII_VERB", regexp_replace(col("ODS_WII_VERB"), "I", "U")) updatedDf.printSchema() val grpbyDF = updatedDf.groupBy("ROW_ID","ODS_WII_VERB").max("STG_LOAD_TS") Sat, 03 Feb 2018 16:45:01 GMT elango_rk 2018-02-03T16:45:01Z