Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Null pointer exception + Spark UDF - Scala

Null pointer exception + Spark UDF - Scala

Rising Star
val returnMatchingWord = (tweetMessage:String) => {
      var word=""
      if(keywordExists(tweetMessage,flagOneKeyWords)){
  println("keyword Exist")
        var matchWord=getMatchingWord(tweetMessage,flagOneKeyWords)
println("Matching Keyword : "+matchWord)
var stringQuery = "select "+matchWord+" from flagOneKeyword"
word = sqlContext.sql(stringQuery).map(x=>x.getString(0)).collect().mkString("")
      }
      word
    }

works fine as a function but when I implement it with my dataframe logic - it throws a null pointer exception. Please advise.

val returnMatchingWordUDF = udf(returnMatchingWord)
val y = x.withColumn("flagOne",returnMatchingWordUDF(lower(col("TweetRawMessage"))))

keyword Exist
keyword Exist
Matching Keyword : xyzhelps
Matching Keyword : xyzsupport
17/05/26 11:11:45 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 12)
java.lang.NullPointerException
        at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:131)
        at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$withPlan(DataFrame.scala:2126)
        at org.apache.spark.sql.DataFrame.select(DataFrame.scala:707)
        at org.apache.spark.sql.DataFrame.select(DataFrame.scala:724)
        at $line158.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:103)
        at $line158.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:97)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
        at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:51)
        at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:49)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)


Don't have an account?
Coming from Hortonworks? Activate your account here