Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to retrieve last 24-hours data from Spark DataFrame using Spark Scala

avatar

I have Timestamp column 2018-08-01 05:02:22.0 , i need to consider or filter the data from the datframe for last 24hrs.

2 REPLIES 2

avatar
Master Guru
@Nikil Katturi

You can use filter function on the data frame

import org.apache.spark.sql.functions._
df.filter(col("<timestamp_column_name>") > lit(date_sub(current_date,1))).show(10,false)

avatar

Thanks that worked