Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to apply a function to a certain columns in the data frame without using withColumn?

Highlighted

How to apply a function to a certain columns in the data frame without using withColumn?

 
1 REPLY 1

Re: How to apply a function to a certain columns in the data frame without using withColumn?

Super Guru

@Venkat

You can use either

.select(expr(function))

(or)

.selectExpr("function")

Example:

val n_df = Seq(("A", 1, "US"), ("AB", 1, "US"), ("ABC", 1, "US")).toDF("city", "num", "country")

.select(expr)

n_df.select(expr("(num * 10) as num"),lit(1).as("op")).show()
+---+---+
|num| op|
+---+---+
| 10|  1|
| 10|  1|
| 10|  1|
+---+---+

.selectExpr:

n_df.selectExpr("num * 10 as num").show(false)
+---+
|num|
+---+
|10 |
|10 |
|10 |
+---+
Don't have an account?
Coming from Hortonworks? Activate your account here