question Re: Sqoop with Oozie error in Archives of Support Questions (Read Only)

Sqoop with Oozie error

Vinod369 — Fri, 16 Sep 2022 10:34:30 GMT

I am running Sqoop action in Oozie, with below script

sqoop import --connect jdbc:sqlserver://SQLM03;database=test --username data_user --password pass1 --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --table table123 -m 1 --target-dir /user/data_user/sqoop

I have placed SQL Server driver jar in Oozie share lib /user/oozie/share/lib/lib_20150429135817/sqoop

I am able to import data using sqoop from command prompt using same script, but getting below error while running from Oozie

 Log Type: stderr
  
  Log Length: 539
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".
  log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  No such sqoop tool: sqoop. See 'sqoop help'.
  Intercepting System.exit(1)
  Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Tried to search it in previous posts, but no solution found.

Please suggest.

Regards,

Vinod

Re: Sqoop with Oozie error

Harsh J — Sun, 14 Aug 2016 22:28:38 GMT

No such sqoop tool: sqoop. See 'sqoop help'.

Your Sqoop Action's commands should begin with just the "import" command, and not include "sqoop" as its first argument, i.e. it should look like this:

<command>import --connect …</command>

And not like this, which is how you've specified it:

<command>sqoop import --connect …</command>

The Sqoop Action of Oozie is documented with an example at http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SqoopActionExtension.html

Re: Sqoop with Oozie error

Vinod369 — Mon, 15 Aug 2016 02:14:49 GMT

Thank you Harsh.

It was my bad to overlook the actul commands inside the Sqoop action.

Now I am getting below error,

Log Type: stderr

Log Length: 3696
log4j:ERROR Could not find value for key log4j.appender.CLA
log4j:ERROR Could not instantiate appender named "CLA".
log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Aug 14, 2016 9:47:55 PM com.microsoft.sqlserver.jdbc.SQLServerConnection <init>
SEVERE: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0.
java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0.

I have below drivers in Oozie sahred lib / user/ oozie/ share/ lib/ lib_20150429135817/ sqoop,

sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar

Also have below drivers,

Edge node Sqoop2: /var/lib/sqoop2 sqljdbc41.jar

Edge node Sqoop: /var/lib/sqoop sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar

Sqoop2 Server: /var/lib/sqoop2 sqljdbc.jar, sqljdbc4.jar, sqljdbc41.jar, sqljdb42.jar

Do I need to add sqljdbc4.jar on Edge node Sqoop2 directory as well remove drivers other than sqljdbc4.jar from all places ? What if in future I want to connect to another SQL Server with different version ?

Are these drivers not come with older version compatibiltiy ?

Thank you,

Vinod

Re: Sqoop with Oozie error

Harsh J — Mon, 15 Aug 2016 07:51:58 GMT

You need just one of those jars in the directory, not all of them.

Just use one, the one named sqljdbc41.jar

(The 4.1 JDBC spec is the latest version supported with JDK7).

Remove everything else (especially remove sqljdbc42.jar which won't work unless you upgrade to JDK8) to avoid this problem.

Re: Sqoop with Oozie error

Vinod369 — Tue, 16 Aug 2016 14:08:35 GMT

I have kept only 1 driver sqljdbc41.jar in all directories and Oozie shared lib path.

But still I am getting below error,

Log Type: stderr
  
  Log Length: 3696
  log4j:ERROR Could not find value for key log4j.appender.CLA
  log4j:ERROR Could not instantiate appender named "CLA".
  log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  Aug 16, 2016 9:45:42 AM com.microsoft.sqlserver.jdbc.SQLServerConnection <init>
  SEVERE: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0.
  java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0.

Even I tried with driver sqljdbc4.jar, but it keep asking other version of driver other than present in directories.

After running a workflow, I realized there is workflow.xml generated in workspace but no job.properties. So I manually created job.properties as below and added in workspace,

Below is workflow.xml generated ,

I am not entering any other job properties or parameters in workflow.

Not sure if I am doing something wrong here, as I can not see jobTracker and nameNode actual names is workflow.xml file.

Please guide in this.

Re: Sqoop with Oozie error

Vinod369 — Tue, 16 Aug 2016 20:41:02 GMT

This issue is resolved now.

Took below steps,

1. Removed oozie.libpath line from job.properties

2. Created lib folder in workspace and placed sqljdbc41.jar in lib folder

So total 3 things placed in workspace, job.properties, workflow.xml and lib folder.

Re: Sqoop with Oozie error

pateljay — Tue, 22 May 2018 11:23:47 GMT

Hi @Vinod369,
I have done same things you mentioned. then I am facing below error.

java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.SqoopMain not found

Please help me to resolve this error.

Regards,

Jay

Re: Sqoop with Oozie error

Tirabassi — Fri, 17 Aug 2018 18:11:03 GMT

I have this same problem, I too tried follow the method of @Vinod369 and not resolved