Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

avatar
Explorer

Here is a snippet of the code:

 

object SparkCSVParser {

def main(args: Array[String]) {

    val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

 

    val restClient = Client.create()
    val webResource = restClient.resource(tsd)

 

// JSONValue works outside of map partition, class is found

 

tokenized.mapPartitions(lines => {
val parser = new CSVParser(',')

    lines.map( line => {
         import org.json.simple.JSONValue

         val columns = parser.parseLine(line)
         var sensorMetrics = ArrayBuffer[String]()


for ((sName, sValuePos) <- nameToPos) {
      val json = ("metric" -> sName) ~
         ("timestamp" -> (hoursInSec * columns(otherNameToPos("HOURS")).toInt).+(today / toSeconds - dayInSec)) ~
         ("value" -> columns(sValuePos)) ~
         ("tags" -> Map {"engine_index" -> columns(otherNameToPos("ENGINE_INDEX"))})

 

        sensorMetrics += pretty(json)
}


    JSONValue.toJSONString(sensorMetrics) // <- it does not work here. Class is nor found
})
}).take(10).foreach(
webResource
.accept(MediaType.APPLICATION_JSON_TYPE)
.`type`(MediaType.APPLICATION_FORM_URLENCODED_TYPE)
.post(classOf[ClientResponse], _))

 

}

 

running on "--master local"

 

Here is the error:

15/01/15 21:43:35 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoClassDefFoundError: org/json/simple/JSONValue
    at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:136)
    at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:119)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)

 

 

How can I make it discover this class?

 

1 ACCEPTED SOLUTION

avatar
Explorer

The problem was indeed in the packaging. I fixed it by including the maven-assembly-plugin.

 

<plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

 

View solution in original post

3 REPLIES 3

avatar
Master Collaborator

Is this library bundled with your app? One guess would be that it is not, and happens to be on the classpath from another dependency on the driver, but is not accidentally found on the executors.

avatar
Explorer

The problem was indeed in the packaging. I fixed it by including the maven-assembly-plugin.

 

<plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

 

avatar
Explorer
Actually, It doesn't work anywhere. I had tested it working within the IDE now when I packaged it up and sent it to the node it did not work even outside the map partition. NVM must be a packaging error