Hey guys, I am badly stuck at running falcon job and trying to solve this since two days.I have designed data pipeline which loads data on hourly basis from HDFS loaction and put it in hive table(table is partitioned already). The Error which I am getting in OOzie UI for falcon process is : Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]. I have checked the oozie launcher logs and yarn logs too but some wrong configuration I didnt able to see yarn logs. What to do in such a case? Where I should look for the error/logs? Where I will get to see complete or string reason of error? Command: yarn logs -applicationId <ID> --------------------------------------------------------------------------------------------------------------------------------------------------------------- There are total 3 XML 1)InputFeedhive.xml(Which refer to HDFS location) 2)processHive.xml(Runs hive script to load data into hive table) 3)hiveOutputFeed.xml(Refer to hive table.) --------------------------------------------------------------------------------------------------------------------------------------------------------------- The hive.hql conatin only: LOAD DATA INPATH "${input}" OVERWRITE INTO TABLE "${falcon_output_table}" PARTITION(${falcon_output_partitions_hive}); --------------------------------------------------------------------------------------------------------------------------------------------------------------- The HDFS Loaction is in following format: /user/manoj/input/bdate=2016-05-21-09/employee.txt --------------------------------------------------------------------------------------------------------------------------------------------------------------- A emloyee.txt contains 12,john,304 13,brown,305 14,jeddy,306 --------------------------------------------------------------------------------------------------------------------------------------------------------------- 1) InputFeedhive.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <feed name="inputFeedHive" xmlns="uri:falcon:feed:0.1"> <frequency>hours(1)</frequency> <timezone>UTC</timezone>
<clusters>
<cluster name="hiveCluster" type="source"> <validity start="2016-05-21T07:00Z" end="2016-05-25T07:00Z"/>
<retention limit="days(1)" action="delete"/>
</cluster> </clusters> <locations>
<location type="data" path="/user/manoj/input/bdate=${YEAR}-${MONTH}-${DAY}-${HOUR}"/> <location type="stats" path="/user/manoj/statsPath${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="meta" path="/user/manoj/metaPath${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
</locations>
<ACL owner="ambari-qa" group="users" permission="0755"/>
<schema location="/none" provider="none"/>
<properties> <property name="jobPriority" value="VERY_HIGH"/>
</properties>
</feed> --------------------------------------------------------------------------------------------------------------------------------------------------------------- 2) hiveOutputFeed.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <feed name="hiveOutputFeed" xmlns="uri:falcon:feed:0.1">
<frequency>hours(1)</frequency>
<timezone>UTC</timezone> <clusters>
<cluster name="hiveCluster" type="source">
<validity start="2016-05-21T07:00Z" end="2016-05-25T07:00Z"/>
<retention limit="days(1)" action="delete"/>
<table uri="catalog:temp:employee#bdate=${YEAR}-${MONTH}-${DAY}-${HOUR}"/> </cluster>
</clusters> <table uri="catalog:temp:employee#bdate=${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<ACL owner="ambari-qa" group="users" permission="0755"/> <schema location="hcat" provider="hcat"/>
</feed>
--------------------------------------------------------------------------------------------------------------------------------------------------------------- 3) processHive.xml <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="processHive" xmlns="uri:falcon:process:0.1"> <tags>new=HiveProcess</tags>
<clusters>
<cluster name="hiveCluster">
<validity start="2016-05-21T07:00Z" end="2016-05-25T07:00Z"/>
</cluster>
</clusters>
<parallel>1</parallel>
<order>FIFO</order>
<frequency>minutes(15)</frequency> <timezone>UTC</timezone>
<inputs> <input name="input" feed="inputFeedHive" start="now(0,0)" end="now(0,0)"/> </inputs>
<outputs>
<output name="output" feed="hiveOutputFeed" instance="now(0,0)"/>
</outputs> <workflow engine="hive" path="/user/manoj/script/hive.hql"/> <retry policy="periodic" delay="minutes(30)" attempts="1"/> <ACL owner="ambari-qa" group="users" permission="0755"/>
</process> --------------------------------------------------------------------------------------------------------------------------------------------------------------- Thanks in advance. Please share you ides with me.
... View more
