Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ERROR running sqoop action from oozie

Solved Go to solution

ERROR running sqoop action from oozie

Contributor

resource-manager-ui.txtHello,

Our HDP version 2.5

When we trying to run sqoop action(to load data from oracle to hive) from oozie we get folowing error in /var/log/oozie/oozie-error.log:

JOB[0000004-161024200820785-oozie-oozi-W] ACTION[0000004-161024200820785-oozie-oozi-W@sqoop] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

And there is nothing more usefull for diagnostic.

Job.properties file listed below:

# properties

nameNode = hdfs://xxxxx:8020

resourceManager = xxxx:8050

queueName=default

oozie.use.system.libpath=true

oozie.wf.application.path = hdfs://xxxxxx:8020/smartdata/oozie/hive_test.xml

mapreduce.framework.name = yarn

When we running this job from command line with "sqoop ..... " as command all working fine.

Please some one tell me how to solve or troubleshoot this.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ERROR running sqoop action from oozie

Contributor

Issue was solved by my self. The solutin was:

1) under folder in which workflow.xml is create folder lib and put there all hive jar files from sharedlibDir(/user/oozie/share/lib/lib_20160928171540)/hive;

2) Create hive-site.xml with contents:

<configuration> <property> <name>ambari.hive.db.schema.name</name> <value>hive</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://xxxxx:9083</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>xxxx:2181,yyyyy:2181,zzzzzz:2181</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/smartdata/hive/</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://xxxxx:5432/hive</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> </configuration>

and put it on hdfs for example in /tmp/hive-site.xml

3) Add following line in workflow.xml:

<file>/tmp/hive-site.xml</file>

This solved my issue.

4 REPLIES 4

Re: ERROR running sqoop action from oozie

New Contributor

Have you tried looking at the job output in the ResourceManager UI or using $ yarn logs -applicationId <application ID>?

Highlighted

Re: ERROR running sqoop action from oozie

Expert Contributor

I believe ShareLib for Sqoop is not registered or missing from Oozie. Check the ooize sharelib by:

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist

if sqoop is missing from the list, copy jars into /user/ooize/share/lib directory and run:

$ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

ref:http://oozie.apache.org/docs/4.1.0/DG_CommandLineTool.html#Getting_list_of_available_sharelib

Re: ERROR running sqoop action from oozie

Contributor

Hello thanks for advice but all fine with shared libraries:

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig

$ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate [ShareLib update status] sharelibDirOld = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 host = http://localhost:11000/oozie sharelibDirNew = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 status = Successful

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig

On Resource manager UI all fine, see attached logs.

Re: ERROR running sqoop action from oozie

Contributor

Issue was solved by my self. The solutin was:

1) under folder in which workflow.xml is create folder lib and put there all hive jar files from sharedlibDir(/user/oozie/share/lib/lib_20160928171540)/hive;

2) Create hive-site.xml with contents:

<configuration> <property> <name>ambari.hive.db.schema.name</name> <value>hive</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://xxxxx:9083</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>xxxx:2181,yyyyy:2181,zzzzzz:2181</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/smartdata/hive/</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://xxxxx:5432/hive</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> </configuration>

and put it on hdfs for example in /tmp/hive-site.xml

3) Add following line in workflow.xml:

<file>/tmp/hive-site.xml</file>

This solved my issue.

Don't have an account?
Coming from Hortonworks? Activate your account here