why Falcon job is Failing?

Manus — Sat, 21 May 2016 14:49:04 GMT

Hey guys,

I am badly stuck at running falcon job and trying to solve this since two days.I have designed data pipeline which loads data on hourly basis from HDFS loaction and put it in hive table(table is partitioned already).

The Error which I am getting in OOzie UI for falcon process is :

Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000].

I have checked the oozie launcher logs and yarn logs too but some wrong configuration I didnt able to see yarn logs.

What to do in such a case?

Where I should look for the error/logs?

Where I will get to see complete or string reason of error?

Command:

yarn logs -applicationId <ID>

---------------------------------------------------------------------------------------------------------------------------------------------------------------

There are total 3 XML

1)InputFeedhive.xml(Which refer to HDFS location)

2)processHive.xml(Runs hive script to load data into hive table)

3)hiveOutputFeed.xml(Refer to hive table.)

---------------------------------------------------------------------------------------------------------------------------------------------------------------

The hive.hql conatin only:

LOAD DATA INPATH "${input}" OVERWRITE INTO TABLE "${falcon_output_table}" PARTITION(${falcon_output_partitions_hive});

---------------------------------------------------------------------------------------------------------------------------------------------------------------

The HDFS Loaction is in following format:

/user/manoj/input/bdate=2016-05-21-09/employee.txt

---------------------------------------------------------------------------------------------------------------------------------------------------------------

A emloyee.txt contains

12,john,304

13,brown,305

14,jeddy,306

---------------------------------------------------------------------------------------------------------------------------------------------------------------

1) InputFeedhive.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<frequency>hours(1)</frequency>

</cluster>

</clusters>

</locations>

</properties>

</feed>

---------------------------------------------------------------------------------------------------------------------------------------------------------------

2) hiveOutputFeed.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<frequency>hours(1)</frequency>

</cluster>

</clusters>

</feed>

---------------------------------------------------------------------------------------------------------------------------------------------------------------

3) processHive.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<tags>new=HiveProcess</tags> <clusters> <cluster name="hiveCluster">

</cluster>

</clusters>

<frequency>minutes(15)</frequency>

</inputs>

</outputs>

</process>

---------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks in advance.

Please share you ides with me.

Re: why Falcon job is Failing?

bandarusridhar1 — Sat, 21 May 2016 19:51:47 GMT

@Manoj Dhake : Can you please share complete error log file? we get "Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]." in many senaerios

Re: why Falcon job is Failing?

KuldeepK — Sun, 22 May 2016 02:07:05 GMT

@Manoj Dhake - Can you please check stderr in Oozie launcher logs?

Re: why Falcon job is Failing?

Manus — Sun, 22 May 2016 14:31:51 GMT

Thanks Sri,

I have checked yarn logs too but about which error log file you are talking?

I mean,oozie error log file/falcon/hive log file?

About whichever log file you are talking,please mention the log file name so that I can immediately send it you.

Re: why Falcon job is Failing?

Manus — Sun, 22 May 2016 14:33:45 GMT

Thank you Kuldeep,

As I am new to falcon and oozie so i dont know where to see Oozie launcher logs?

At which location?

Re: why Falcon job is Failing?

Manus — Mon, 23 May 2016 15:46:03 GMT

Hello guys,

I have resolved this problem.Actually There was problem in hive.hql file

Here is Correct Hive script:

LOAD DATA INPATH '${input}/employee.txt' OVERWRITE INTO TABLE temp.employee PARTITION(${falcon_output_partitions_hive});

Re: why Falcon job is Failing?

writetoaditikum — Fri, 07 Oct 2016 12:54:55 GMT

@Manoj Dhake :- Hi Manoj, finally the issue has got resolved ? if so , can you share the workaround. Thanks .

Re: why Falcon job is Failing?

writetoaditikum — Fri, 07 Oct 2016 13:11:41 GMT

@Manoj Dhake

@Kuldeep Kulkarni

I replicated the same scenario and getting same error of Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]. Detailed error from STDERR is

Logging initialized using configuration in /yarn1/hadoop/yarn/local/usercache/id847257/appcache/application_1475673186013_1313/container_e69_1475673186013_1313_01_000002/hive-log4j.properties
FAILED: IllegalArgumentException java.net.URISyntaxException: Illegal character in scheme name at index 0: 2016-10-07,hdfs:
Intercepting System.exit(40000)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]

I looked for any blank character at the given index, but didn't find any resolution.

question Re: why Falcon job is Failing? in Support Questions

why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?

Re: why Falcon job is Failing?