Support Questions
Find answers, ask questions, and share your expertise

Error while invoking the Java value object class in Spark (using scala)

Error while invoking the Java value object class in Spark (using scala)

New Contributor

Hi,

Can some one please help me to resolve my issue ? Below are the details.

I am wanted to use Java bean class as a variable in Scala case class. While applying the map during RDD transformation, I am getting error like

Exception in thread "main" java.lang.UnsupportedOperationException: No Encoder found for com.java.training.EmployeeVO - field (class: "com.java.training.EmployeeVO", name: "emp") - root class: "com.basic.Employee" at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$serializerFor(ScalaReflection.scala:598) at org.apache.spark.sql.catalyst.ScalaReflection$anonfun$9.apply(ScalaReflection.scala:592) at org.apache.spark.sql.catalyst.ScalaReflection$anonfun$9.apply(ScalaReflection.scala:583) at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.immutable.List.flatMap(List.scala:344) at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$serializerFor(ScalaReflection.scala:583) at org.apache.spark.sql.catalyst.ScalaReflection$.serializerFor(ScalaReflection.scala:425) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:61) at org.apache.spark.sql.Encoders$.product(Encoders.scala:274) at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:47) at com.basic.ScalaJavaTest$.main(ScalaJavaTest.scala:26) at com.basic.ScalaJavaTest.main(ScalaJavaTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Please find my scala source code below. EmployeeVO is my java class from external java jar dependency.

package com.basic

import org.apache.spark.sql.{Row, SparkSession}
import com.java.training.EmployeeVO

case class Employee(var emp:EmployeeVO )

object ScalaJavaTest {

  def main(args: Array[String])={
    System.setProperty("hadoop.home.dir", "C:\\hadoop\\");

    val spark = SparkSession.builder.master("local")
      .appName("my-spark-app")
      .config("spark.sql.warehouse.dir", "file:///C:/Users/Sundar/IdeaProjects/MySparkProj/spark-warehouse")
      .getOrCreate()

    import spark.implicits._

    val emp= spark.read.json("user/data/employee.json")

    val test=  emp.map(row=>populteEmployee(row))
    test.show()
  }

  def populteEmployee(row:Row):Employee = {
    println("populteEmployee() Starting")
    var empVO:EmployeeVO = new EmployeeVO()

    empVO.setName(row.getString(1))
    empVO.setCity(row.getString(0))
    empVO.setState(row.getString(2))

    return Employee(empVO)
  }

}