Is it possible to write into a location directly instead of hdfs path ? ... View more

Sure. One way I can think of achieving this is by creating a UDF using random and calling the udf within withColumn using coalesce. See below:    scala> df1.show() +----+--------+----+ | id| name| age| +----+--------+----+ |1201| satish|39 | |1202| krishna|null| << |1203| amith|47 | |1204| javed|null| << |1205| prudvi|null| << +----+--------+----+ scala> val arr = udf(() => scala.util.Random.nextInt(10).toString()) scala> val df2 = df1.withColumn("age", coalesce(df1("age"), arr())) scala> df2.show() +----+--------+---+ | id| name|age| +----+--------+---+ |1201| satish| 39| |1202| krishna| 2 | << |1203| amith| 47| |1204| javed| 9 | << |1205| prudvi| 7 | << +----+--------+---+   ... View more