question Re: ERROR running sqoop action from oozie in Archives of Support Questions (Read Only)

ERROR running sqoop action from oozie

RAkhmadeev — Wed, 26 Oct 2016 19:47:23 GMT

Our HDP version 2.5

When we trying to run sqoop action(to load data from oracle to hive) from oozie we get folowing error in /var/log/oozie/oozie-error.log:

JOB[0000004-161024200820785-oozie-oozi-W] ACTION[0000004-161024200820785-oozie-oozi-W@sqoop] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

And there is nothing more usefull for diagnostic.

Job.properties file listed below:

# properties

nameNode = hdfs://xxxxx:8020

resourceManager = xxxx:8050

queueName=default

oozie.use.system.libpath=true

oozie.wf.application.path = hdfs://xxxxxx:8020/smartdata/oozie/hive_test.xml

mapreduce.framework.name = yarn

When we running this job from command line with "sqoop ..... " as command all working fine.

Please some one tell me how to solve or troubleshoot this.

Re: ERROR running sqoop action from oozie

twilson — Wed, 26 Oct 2016 22:44:47 GMT

Have you tried looking at the job output in the ResourceManager UI or using $ yarn logs -applicationId <application ID>?

Re: ERROR running sqoop action from oozie

umair_khan — Thu, 27 Oct 2016 00:37:22 GMT

I believe ShareLib for Sqoop is not registered or missing from Oozie. Check the ooize sharelib by:

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist

if sqoop is missing from the list, copy jars into /user/ooize/share/lib directory and run:

$ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate

ref:http://oozie.apache.org/docs/4.1.0/DG_CommandLineTool.html#Getting_list_of_available_sharelib

Re: ERROR running sqoop action from oozie

RAkhmadeev — Thu, 27 Oct 2016 16:25:38 GMT

Hello thanks for advice but all fine with shared libraries:

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig

$ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate [ShareLib update status] sharelibDirOld = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 host = http://localhost:11000/oozie sharelibDirNew = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 status = Successful

$ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig

On Resource manager UI all fine, see attached logs.

Re: ERROR running sqoop action from oozie

RAkhmadeev — Fri, 28 Oct 2016 16:38:49 GMT

Issue was solved by my self. The solutin was:

1) under folder in which workflow.xml is create folder lib and put there all hive jar files from sharedlibDir(/user/oozie/share/lib/lib_20160928171540)/hive;

2) Create hive-site.xml with contents:

<configuration> <property> <name>ambari.hive.db.schema.name</name> <value>hive</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://xxxxx:9083</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>xxxx:2181,yyyyy:2181,zzzzzz:2181</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/smartdata/hive/</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://xxxxx:5432/hive</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> </configuration>

and put it on hdfs for example in /tmp/hive-site.xml

3) Add following line in workflow.xml:

This solved my issue.