java.lang.NoClassDefFoundError: org/json/simple/JSONValue inside map partition

[#1]

Here is a snippet of the code:

object SparkCSVParser {

def main(args: Array[String]) {

    val sc = new SparkContext(new SparkConf().setAppName("Spark Count"))

    val restClient = Client.create()
    val webResource = restClient.resource(tsd)

// JSONValue works outside of map partition, class is found

tokenized.mapPartitions(lines => {
val parser = new CSVParser(',')

    lines.map( line => {
         import org.json.simple.JSONValue

         val columns = parser.parseLine(line)
         var sensorMetrics = ArrayBuffer[String]()

for ((sName, sValuePos) <- nameToPos) {
      val json = ("metric" -> sName) ~
         ("timestamp" -> (hoursInSec * columns(otherNameToPos("HOURS")).toInt).+(today / toSeconds - dayInSec)) ~
         ("value" -> columns(sValuePos)) ~
         ("tags" -> Map {"engine_index" -> columns(otherNameToPos("ENGINE_INDEX"))})

        sensorMetrics += pretty(json)
}

    JSONValue.toJSONString(sensorMetrics) // <- it does not work here. Class is nor found
})
}).take(10).foreach(
webResource
.accept(MediaType.APPLICATION_JSON_TYPE)
.`type`(MediaType.APPLICATION_FORM_URLENCODED_TYPE)
.post(classOf[ClientResponse], _))

}

running on "--master local"

Here is the error:

15/01/15 21:43:35 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoClassDefFoundError: org/json/simple/JSONValue
    at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:136)
    at com.decisioniq.spark.SparkTrainCSVParser$$anonfun$main$1$$anonfun$apply$1.apply(SparkTrainCSVParser.scala:119)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)

How can I make it discover this class?

[#2]

Is this library bundled with your app? One guess would be that it is not, and happens to be on the classpath from another dependency on the driver, but is not accidentally found on the executors.

[#3 · Accepted solution]

The problem was indeed in the packaging. I fixed it by including the maven-assembly-plugin.

<plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.5.3</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id> <!-- this is used for inheritance merges -->
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

[#4]

Actually, It doesn't work anywhere. I had tested it working within the IDE now when I packaged it up and sent it to the node it did not work even outside the map partition. NVM must be a packaging error