question Oozie fails with: Oozie submit sqoop which execute hive script in Support Questions

Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Mon, 28 Mar 2016 13:08:53 GMT

Hi:

I used oozie execute a sqoop script, which import mysql data to hive table. But the program is error.

the main of "workflow.xml"

<action name="sqoop-node"> <sqoop xmlns="uri:oozie:sqoop-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <command>import --connect jdbc:mysql://xxxx:3306/xxxx --username xxx--password xxx --table area --hive-import</command> </sqoop>

The program have save the mysql data to hdfs，but can not import to hive.

The log said :

ERROR org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Hive exited with status 1 at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:394) at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:344) at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:245) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:148) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235) at org.apache.sqoop.Sqoop.main(Sqoop.java:244) at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

But I can execute under the shell mode.

sqoop import --connect jdbc:mysql://xxxx:3306/xxxx --username xxx--password xxx --table area --hive-import

is ok.

Anyone know how to solute this problem ?

Re: Oozie fails with: Oozie submit sqoop which execute hive script

bhagan — Tue, 29 Mar 2016 01:07:44 GMT

Can you verify if you have added the hive-site.xml to HDFS and included a reference to that file in your workflow? I don't see it referenced in the the sqoop action.

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Tue, 29 Mar 2016 09:19:05 GMT

Thanks for your response.

I use the HDP2.3 and don't change the config file.

the hive-site.xml include the hdfs

But I don't know how to add reference file to my workflow.

Can you tell me?

--------------------------------------------------

<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf">

<job-tracker>${jobTracker}</job-tracker>

<name-node>${nameNode}</name-node>

<name>mapred.job.queue.name</name>

<value>${queueName}</value>

</property>

</configuration>

<command>import --hive-import --connect jdbc:mysql://xxxxxx:3306/xxxx --username root --password xxxxx --table area</command>

</sqoop>

</action>

<message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

</kill>

</workflow-app>

--------------------------------------------------

Re: Oozie fails with: Oozie submit sqoop which execute hive script

KuldeepK — Tue, 29 Mar 2016 10:14:44 GMT

@allen huang

you can add hive-site.xml location(hdfs) under <file> tag in your sqoop action.

            <strong><file>/tmp/hive-site.xml#hive-site.xml</file></strong>

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Tue, 29 Mar 2016 12:32:36 GMT

I have added hive-site.xml to the directory of workflow.xml add <file>hive-site.xml#hive-site.xml</file> to workflow.xml, I get the same errors.

So what can I do next?

-----------------------------------------------------

<job-tracker>${jobTracker}</job-tracker>

<name-node>${nameNode}</name-node>

</prepare>

<name>mapred.job.queue.name</name>

<value>${queueName}</value> </property>

</configuration>

<command>import --hive-import --connect jdbc:mysql://xxxx:3306/xxxx --username xx --password xxxx --table area</command>

</sqoop>

</action>

<message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

</kill>

</workflow-app>

-----------------------------------------------------

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Tue, 29 Mar 2016 15:47:56 GMT

Oh, This question have been solved. I forgot to input packages under the hive_home/lib.

But, I found another question.

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

The Log said:

-------------------------------------------------------

2016-03-29 16:32:29,564 WARN  [main] util.MRApps (MRApps.java:parseDistributedCacheArtifacts(610)) - cache file (mapreduce.job.cache.files) hdfs://cluster1.new/user/root/share/lib/sqoop/mysql-connector-java-5.1.38-bin.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://cluster1.new/user/root/.staging/job_1459235663418_0020/libjars/mysql-connector-java-5.1.38-bin.jar This will be an error in Hadoop 2.0
2016-03-29 16:32:29,801 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted application application_1459235663418_0020
2016-03-29 16:32:29,823 INFO  [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://cluster2.new:8088/proxy/application_1459235663418_0020/
2016-03-29 16:32:29,824 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job: job_1459235663418_0020
2016-03-29 16:32:33,869 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_1459235663418_0020 running in uber mode : false
2016-03-29 16:32:33,870 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) -  map 0% reduce 0%
2016-03-29 16:32:38,898 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) -  map 100% reduce 0%
2016-03-29 16:32:39,904 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1378)) - Job job_1459235663418_0020 completed successfully
2016-03-29 16:32:39,938 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 30
	File System Counters
		FILE: Number of bytes read=0
		FILE: Number of bytes written=1377552
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=545
		HDFS: Number of bytes written=1090
		HDFS: Number of read operations=16
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=8
	Job Counters 
		Launched map tasks=4
		Other local map tasks=4
		Total time spent by all maps in occupied slots (ms)=8075
		Total time spent by all reduces in occupied slots (ms)=0
		Total time spent by all map tasks (ms)=8075
		Total vcore-seconds taken by all map tasks=8075
		Total megabyte-seconds taken by all map tasks=16537600
	Map-Reduce Framework
		Map input records=51
		Map output records=51
		Input split bytes=545
		Spilled Records=0
		Failed Shuffles=0
		Merged Map outputs=0
		GC time elapsed (ms)=160
		CPU time spent (ms)=3120
		Physical memory (bytes) snapshot=923926528
		Virtual memory (bytes) snapshot=14719983616
		Total committed heap usage (bytes)=886046720
	File Input Format Counters 
		Bytes Read=0
	File Output Format Counters 
		Bytes Written=1090
14714 [main] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Transferred 1.0645 KB in 11.9988 seconds (90.8424 bytes/sec)
2016-03-29 16:32:39,944 INFO  [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(184)) - Transferred 1.0645 KB in 11.9988 seconds (90.8424 bytes/sec)
14716 [main] INFO  org.apache.sqoop.mapreduce.ImportJobBase  - Retrieved 51 records.
2016-03-29 16:32:39,946 INFO  [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(186)) - Retrieved 51 records.
14727 [main] INFO  org.apache.sqoop.manager.SqlManager  - Executing SQL statement: SELECT t.* FROM `area` AS t LIMIT 1
2016-03-29 16:32:39,957 INFO  [main] manager.SqlManager (SqlManager.java:execute(757)) - Executing SQL statement: SELECT t.* FROM `area` AS t LIMIT 1
14732 [main] INFO  org.apache.sqoop.hive.HiveImport  - Loading uploaded data into Hive
2016-03-29 16:32:39,962 INFO  [main] hive.HiveImport (HiveImport.java:importTable(195)) - Loading uploaded data into Hive
Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

------------------------------------------------------------

I don't know how to find the error place.

The log didn't say anything except "Intercepting System.exit(1)".

Had anyone encountered this problem?

Re: Oozie fails with: Oozie submit sqoop which execute hive script

aervits — Tue, 29 Mar 2016 15:53:18 GMT

Please refer to this article to troubleshoot Oozie workflows https://community.hortonworks.com/content/kbentry/9148/troubleshooting-an-oozie-flow.html

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Wed, 30 Mar 2016 08:08:22 GMT

Please to see the comment that I have responded to Kuldeep Kulkarni. I have added the hive-site.xml

<name>mapred.job.queue.name</name>

<value>${queueName}</value> </property>

</configuration>

<command>import --hive-import --connect jdbc:mysql://xxxx:3306/xxxx --username xx --password xxxx --table area</command>

<file>hive-site.xml#hive-site.xml</file>

</sqoop>

Now, I encounter the question "InterceptingSystem.exit(1)".

Can you help me?

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Wed, 30 Mar 2016 08:20:00 GMT

Oh, you give me the URL which teach me how to find the error log.

But my question is :

I found the error log, but I couldn't locate the question from the error log.

The log said:

--------------------------------------------------

14732[main] INFO org.apache.sqoop.hive.HiveImport-Loading uploaded data intoHive

2016-03-2916:32:39,962 INFO [main] hive.HiveImport(HiveImport.java:importTable(195))-Loading uploaded data intoHive

InterceptingSystem.exit(1)

<<<Invocation of Mainclass completed <<<

FailingOozieLauncher,Mainclass[org.apache.oozie.action.hadoop.SqoopMain],exit code [1]

-------------------------------------------------------------------------

The job had finished and didn't say any error information, but it said "InterceptingSystem.exit(1)" finally, I don't know why the job failed.

Re: Oozie fails with: Oozie submit sqoop which execute hive script

KuldeepK — Mon, 04 Apr 2016 00:17:37 GMT

@allen huang - If you know the application id then you can get logs via command line which would give you some clue.

yarn logs -applicationId <application-id>

Re: Oozie fails with: Oozie submit sqoop which execute hive script

fengyeyouni — Tue, 05 Apr 2016 08:01:07 GMT

Thanks,

I have solved this problem.

Refer to :

https://community.hortonworks.com/questions/25121/oozie-execute-sqoop-falls.html#answer-25290

Re: Oozie fails with: Oozie submit sqoop which execute hive script

skhaksho — Sat, 17 Feb 2018 02:48:04 GMT

Hi @allen huang. I faced the same issue. so you solved the problem by removing some of the properties of hive-site.xml? Do you know which properties were making the problem?

@Kuldeep Kulkarni, @Artem Ervits do you guys have any idea in this regards?