question Re: java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition in Archives of Support Questions (Read Only)

java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

gmortiz17 — Fri, 16 Sep 2022 09:19:22 GMT

Here is a snippet of the code:

object SparkCSVParser {

def main(args: Array[String]) {

val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

val restClient = Client.create()
val webResource = restClient.resource(tsd)

// JSONValue works outside of map partition, class is found

tokenized.mapPartitions(lines => {
val parser = new CSVParser(',')

lines.map( line => {
import org.json.simple.JSONValue

val columns = parser.parseLine(line)
var sensorMetrics = ArrayBuffer[String]()

for ((sName, sValuePos) <- nameToPos) {
      val json = ("metric" -> sName) ~
         ("timestamp" -> (hoursInSec * columns(otherNameToPos("HOURS")).toInt).+(today / toSeconds - dayInSec)) ~
         ("value" -> columns(sValuePos)) ~
         ("tags" -> Map {"engine_index" -> columns(otherNameToPos("ENGINE_INDEX"))})

sensorMetrics += pretty(json)
}

JSONValue.toJSONString(sensorMetrics) // <- it does not work here. Class is nor found
})
}).take(10).foreach(
webResource
.accept(MediaType.APPLICATION_JSON_TYPE)
.`type`(MediaType.APPLICATION_FORM_URLENCODED_TYPE)
.post(classOf[ClientResponse], _))

}

running on "--master local"

Here is the error:

15/01/15 21:43:35 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoClassDefFoundError: org/json/simple/JSONValue
   at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:136)
   at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:119)
   at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
   at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
   at scala.collection.Iterator$class.foreach(Iterator.scala:727)

How can I make it discover this class?

Re: java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

srowen — Fri, 16 Jan 2015 03:53:39 GMT

Is this library bundled with your app? One guess would be that it is not, and happens to be on the classpath from another dependency on the driver, but is not accidentally found on the executors.

Re: java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

gmortiz17 — Fri, 16 Jan 2015 03:55:23 GMT

Actually, It doesn't work anywhere. I had tested it working within the IDE now when I packaged it up and sent it to the node it did not work even outside the map partition. NVM must be a packaging error

Re: java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

gmortiz17 — Fri, 23 Jan 2015 19:14:00 GMT

The problem was indeed in the packaging. I fixed it by including the maven-assembly-plugin.

<plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>