I have a dataframe in which a particular integer column contains null value. I wanted to replace all null value in this column with -1 value so tried to use following code :
val df5= df4.na.fill(-1 ,Seq("SEQ-NO"))
On HDP 2.6 with Spark 2.1 ,following exception is thrown:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameNaFunctions.fill(JLscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;
Just to verify this issue, I downloaded spark 2.1.1 from https://spark.apache.org/downloads.html and executed same program. This time it works fine.
Any idea how to solve this issue on HDP 2.6 with Spark 2.1 ?
I tried to reproduce your issue on HDP 2.6 but unfortunately couldn't. However, I am pretty sure that Spark 2.1.1 fixes your problem, and HDP 2.6.1 is shipped with Spark 2.1.1 (HDP 2.6.0 has Spark 2.1.0 which might have this error). Is it a viable option for you to upgrade?