Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
DataFrame join with OR condition
Labels:
- Labels:
-
Apache Spark
Explorer
Created ‎02-25-2017 10:25 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have the following join which is making my spark application hang here and never produces the result. Is OR condition on supported in Spark Dataframes?
DataFrame DFJoin = DF1.join(DF2, DF1.col("device").equalTo(DF2.col("id")).or(DF1.col("device").equalTo(DF2.col("new_id"))), "inner");
2 REPLIES 2
Master Mentor
Created ‎02-25-2017 10:34 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can use with filter
df2 = df1.filter($"Status" === 2 || $"Status" === 3)
Explorer
Created ‎02-26-2017 02:19 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks. The application is in Java, so what is going wrong above in the join statement?
