Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

get a value of the date that corresponds to a specific range of date

get a value of the date that corresponds to a specific range of date

Explorer

Hello guys , i am new to spark and scala. i have two dataframes i want to join them in order to get a value based on where a date is situtaed among a set of start and end date .Here is what i did as example.Thanks i advance

Here is an example

val histocaisse = spark.read.option("header", true).csv("C:\\Users\\MHT\\Desktop\\histocaisse.csv")
.withColumn("pos_id",'pos_id.cast(LongType))
.withColumn("DEBUT",'DEBUT.cast(DateType))
.withColumn("FIN",'FIN.cast(DateType))
.withColumn("PRIXV",'PRIXV.cast(DoubleType))



val test = spark.read.option("header", true).csv("C:\\Users\\MHT\\Desktop\\test.csv")
.withColumn("synch_date",'synch_date.cast(DateType))
test.show

val bis = spark.read.option("header", true).csv("C:\\Users\\MHT\\Desktop\\bis.csv")

.withColumn("synch_date",'synch_date.cast(DateType))
.select("pos_id","synch_date","sale_price") +------+----------+----------+-----+ |pos_id| DEBUT| FIN|PRIXV| +------+----------+----------+-----+ | 1|2000-01-02|2000-01-07| 2.5| | 2|2000-02-18|2000-02-25| 2.3| | 3|2000-03-02|2000-03-25| 14.0| +------+----------+----------+-----+ +------+----------+ |pos_id|synch_date| +------+----------+ | 1|2000-01-03| | 2|2000-02-19| | 3|2000-03-20| +------+----------+ +------+----------+----------+ |pos_id|synch_date|sale_price| +------+----------+----------+ | 1|2000-01-03| 2.5| | 2|2000-02-19| 2.3| | 3|2000-03-20| 14| +------+----------+----------+
Don't have an account?
Coming from Hortonworks? Activate your account here