question Re: Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject in Support Questions

Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject

Howchoy — Fri, 08 Jul 2016 05:51:46 GMT

Command to submit the job:

hadoop jar driver-collection-1.0-SNAPSHOT.jar multipleoutputjson /test/inputjson/json_input.txt /test/outputjson

Error: java.lang.ClassNotFoundException: org.json.JSONObject, see attached.error-message.txt

I have below dependency for org.json.JSONObject (filename: json-20160212.jar) in pom.xml:

<dependency> 
<groupId>org.json</groupId> 
<artifactId>json</artifactId> 
<version>20160212</version> 
<scope>compile</scope> 
</dependency>

Code: The codes is pretty much like this.

I have tried the following links but in vain. Please advise your insight.

Re: Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject

Howchoy — Fri, 08 Jul 2016 09:32:12 GMT

I used the method - "Distributed cache" from "Hadoop: Add third-party libraries to MapReduce job" and got it working.

The following is my steps:

Copied json-20160212.jar to /user/root/lib/json-20160212.jar in hdfs.
Added the following codes in Driver class:
- job.addCacheFile(new Path("/user/root/lib/json-20160212.jar").toUri());
- job.setJarByClass(JSONObject.class);
Compiled the codes and ran the test.

Still very anxious to learn to solve it by using the following methods:

Add libjars option
Add jar files to Hadoop classpath
Create a fat jar

Thank you.

Re: Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject

aervits — Fri, 08 Jul 2016 10:16:00 GMT

@Charles Chen

I don't see an issue with your steps, what I remember from past experiences working with org.json is that some classes were refactored out and perhaps org.json.JSONOBject is not available in version 20160212 try earlier version like 20140107. This will create a fat jar, <scope> is optional as compile is default action. That or -libjars are viable options, let us know if you're still having issues, I'll try to help.

Re: Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject

Howchoy — Sat, 09 Jul 2016 09:28:26 GMT

I have tried 20160212 and 20090211, and only got it working for Distributed cache. I got the jar from below link:

https://mvnrepository.com/artifact/org.json/json

Below is my command. How do I use -libjars?

hadoop jar driver-collection-1.0-SNAPSHOT.jar multipleoutputjson /test/inputjson/json_input.txt /test/outputjs

Thank you for your reply.

Re: Add third-party lib to MapReduce job Error: java.lang.ClassNotFoundException: org.json.JSONObject

aervits — Tue, 12 Jul 2016 16:04:25 GMT

Please see this for example https://dzone.com/articles/using-libjars-option-hadoop

And http://stackoverflow.com/questions/6890087/problem-with-libjars-in-hadoop