Support Questions

Find answers, ask questions, and share your expertise

How to retrieve last 24-hours data from Spark DataFrame using Spark Scala

I have Timestamp column 2018-08-01 05:02:22.0 , i need to consider or filter the data from the datframe for last 24hrs.

2 REPLIES 2

Super Guru
@Nikil Katturi

You can use filter function on the data frame

import org.apache.spark.sql.functions._
df.filter(col("<timestamp_column_name>") > lit(date_sub(current_date,1))).show(10,false)

Thanks that worked

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.