Sqoop import exception java.lang.NoClassDefFoundError: org/json/JSONObject

[#1]

Hello There,

I have configured CDH5.10.0 (Manual configuration mode using YUM) on CentOS 7. It is a Pseudo distributed mode for learning purpose.

I have opted for Sqoop 1 and below is the sqoop version:

Sqoop 1.4.6-cdh5.10.0

When trying to import table from MySQL (mysql  Ver 14.14 Distrib 5.7.17, for Linux (x86_64) using  EditLine wrapper) I'm getting below java exception:

"Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject"

I tried to execute few other sqoop commands like list-databases, list-tables, eval. All these commands returned expected results without any errors.

below is sqoop statement and the error message:

$ sqoop import --connect jdbc:mysql://localhost/employees --table employees --username xxx -P
Warning: /usr/lib/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/02/12 17:44:39 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.10.0
Enter password:
17/02/12 17:44:43 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/02/12 17:44:43 INFO tool.CodeGenTool: Beginning code generation
Sun Feb 12 17:44:43 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
17/02/12 17:44:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employees` AS t LIMIT 1
17/02/12 17:44:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employees` AS t LIMIT 1
17/02/12 17:44:44 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-yj/compile/50d8bdd75972961328544c90d3d8d539/employees.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/02/12 17:44:45 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yj/compile/50d8bdd75972961328544c90d3d8d539/employees.jar
17/02/12 17:44:45 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/02/12 17:44:45 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/02/12 17:44:45 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/02/12 17:44:45 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/02/12 17:44:45 INFO mapreduce.ImportJobBase: Beginning import of employees
17/02/12 17:44:45 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
    at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
    at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:767)
    at org.apache.sqoop.mapreduce.JobBase.putSqoopOptionsToConfiguration(JobBase.java:388)
    at org.apache.sqoop.mapreduce.JobBase.createJob(JobBase.java:374)
    at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:256)
    at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
    at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 15 more

I have spent 3-4 hours searching for solution in community topics as well some other blogs, but could resolve the problem.

Request you all experts to please guide how to get rid of this exception.

Thanks in advance!

Best Regards,

YJ

[#2 · Accepted solution]

By spending some more time reading blogs:

I found solution as below:

downloaded java-json.jar file from location http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm

stored this jar file at location /usr/lib/sqoop/lib/java-json.jar

But I wonder, if JSONObject class is required to process sqoop import, why it is not found in any json jar found under hadoop classpath or anyother component classpath?

when searched for available json jar files I found below list

./usr/lib/hadoop/lib/jersey-json-1.9.jar
./usr/lib/hadoop-yarn/lib/jersey-json-1.9.jar
./usr/lib/hadoop-mapreduce/jersey-json-1.9.jar
./usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar
./usr/lib/hive/lib/metrics-json-3.0.2.jar
./usr/lib/sentry/lib/jersey-json-1.9.jar
./usr/lib/sentry/lib/metrics-json-3.0.2.jar
./usr/lib/kite/lib/kite-morphlines-json-1.0.0-cdh5.10.0.jar
./usr/lib/kite/lib/metrics-json-3.0.2.jar

Any expert advise appreciated.

Thank you!

[#3]

Jersey is basically a RESTful Web Services framework . Jersey json  jar usess another piece of code to convert to /  from Object from Json .

JSONObject  on the other hand is a class used to decode the Json object .  Each jar have their own set of implentation for class , interface , abstard method etc it  should be lossely coupled . Thats the reason we are not able to find JsonObject class in those jars . Different implementation and purpose. 

[#4]

@csgunaThank you for information 🙂

but I'm not convinced, why java-json.jar was missing? I believe, I would have missed any step while installing/configuring sqoop.

After adding java-json.jar my problem have been solved.

[#5]

This solution dinn work for my 5.10 quick start. I solved by copying to below location.

download java-json.jar and copy to  /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/

sudo cp /home/cloudera/Downloads/java-json.jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/

[#6]

Hi.. I am also facing similar kind of problem when I am trying to use sqoop job --create function and below is the command and error I am facing. 

[hdfs@master root]$ sqoop job --create shortcut -- import --connect jdbc:mysql://master.cluster.com:3306/hadoop_example?zeroDateTimeBehavior=convertToNull --username root --table school --driver com.mysql.jdbc.Driver --incremental append --last-value '2017-05-29 02:02:51.0' --check-column timestamp -m 1
Warning: /usr/hdp/2.5.0.0-1245/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/hdp/2.5.0.0-1245//sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/07/03 00:17:10 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:42)
at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:759)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.createInternal(HsqldbJobStorage.java:399)
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.create(HsqldbJobStorage.java:379)
at org.apache.sqoop.tool.JobTool.createJob(JobTool.java:181)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:294)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.main(Sqoop.java:243)
Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more

I tried to follow your suggestion to download the java-json.jar file and placed it /usr/lib/sqoop/lib/java-json.jar.(in my system only /usr/lib is avaliable. I manually created 2 folder /sqoop/lib.) 

Please let me know if you can help me. 

[#7]

nah 

Check the repo yum whether you have installed Sqoop client .

sudo yum install sqoop

if not create one . You dont have to create those folders . mate

side note  anytime when you place any jar maually check the permission and ownershipp . 

[#8]

thank you for your help. I installed sqoop client by using AMBARI-SERVER. I am able to run all Sqoop commands normally without any issue and I installed the sqoop again as you suggested. But still no luck..

[#9]

I had this problem as well with sqoop and found a way to solve this without downloading a jar from somewhere in the internet.

Sqoop tries to get the classpath from several scripts (hbase classpath, hcat -classpath, accumulo classpath) and adds them to the own classpath. As it seems sqoop is not complaining that it can not find hcat and so it skips it silently.

I solved this problem with installing hive-hcatalog and now sqoop adds the hive classpath over hcat -classpath as well and can use the JSONObject class that is included in hive-exec.jar (CDH5.11).

[#10]

sudo cp /home/cloudera/Downloads/java-json.jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/ 

Above solution worked for me too. Thank you