Support Questions
Find answers, ask questions, and share your expertise

dataframe.na.fill for integer value is throwing exception

Hello,

I have a dataframe in which a particular integer column contains null value. I wanted to replace all null value in this column with -1 value so tried to use following code :

val df5= df4.na.fill(-1 ,Seq("SEQ-NO"))

On HDP 2.6 with Spark 2.1 ,following exception is thrown:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameNaFunctions.fill(JLscala/collection/Seq;)Lorg/apache/spark/sql/Dataset;

Just to verify this issue, I downloaded spark 2.1.1 from https://spark.apache.org/downloads.html and executed same program. This time it works fine.

Any idea how to solve this issue on HDP 2.6 with Spark 2.1 ?

5 REPLIES 5

Expert Contributor
@brijesh gautam

I tried to reproduce your issue on HDP 2.6 but unfortunately couldn't. However, I am pretty sure that Spark 2.1.1 fixes your problem, and HDP 2.6.1 is shipped with Spark 2.1.1 (HDP 2.6.0 has Spark 2.1.0 which might have this error). Is it a viable option for you to upgrade?

Thanks @gnovak . Is there any upgrade path available for updating HDP from 2.6.0 to 2.6.1 ?

Expert Contributor

Thanks @gnovak . Please let me source for downloading HDP 2.6.1 So I can perform upgrade from HDP 2.6.0 to HDP 2.6.1.

Super Collaborator

hi @brijesh gautam,

coalesce could do same what you are after.

sql("""select COALESCE(col1,-1),col2 from test""").show
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.