Member since
09-29-2015
57
Posts
49
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1477 | 05-25-2017 06:03 PM | |
1332 | 10-19-2016 10:17 PM | |
1644 | 09-28-2016 08:41 PM | |
985 | 09-21-2016 05:46 PM | |
4597 | 09-06-2016 11:49 PM |
01-31-2016
08:16 PM
2 Kudos
@Nayan Paul: Your process xml has the validity as <validity start="2015-12-01T23:33Z" end="2018-01-03T23:33Z"/> and frequency is every 5 minutes. Instance Number | Process Instance start Time | Feeds to process[currentMonth(0,0,0)-currentMonth(31,0,0)]
1 2015-12-01T23:33Z 2015-12-01T00:00Z - 2015-12-31T00:00Z
2 2015-12-01T28:33Z 2015-12-01T00:00Z - 2015-12-31T00:00Z
3 2015-12-01T33:33Z 2015-12-01T00:00Z - 2015-12-31T00:00Z
... As you can see, process instances generated are for 2015-12 and feeds generated in 2016-01 will be processed when the process instance start time is 2016-01-*. Please change the process instance validity start in process xml to "2016-01-01T00:00Z". Thanks!
... View more
01-29-2016
06:14 PM
1 Kudo
Nayan Paul: Can you provide the input feed and process entities used when the error was thrown? Falcon throws this error if process input feed start is before the input feed validity start. currentMonth(day,hour,minute): Current month takes the reference to start of the month with respect to process instance start time. One thing to keep in mind is that day is added to the first day of the month. So the value of day is the number of days you want to add to the first day of the month. For example: for instance start time 2010-01-12T01:30Z and El as currentMonth(3,2,40) will correspond to feed created at 2010-01-04T02:40Z and currentMonth(0,0,0) will mean 2010-01-01T00:00Z. Looks like currentMonth is evaluating date which is before input feed validity start.
... View more
01-28-2016
10:20 PM
3 Kudos
Benjamin Leonhardi: You may be hitting this issue FALCON-1647. What HDP version are you using? Thanks!
... View more
01-28-2016
09:30 PM
Benjamin Leonhardi: Can you please provide the error you are getting? Also, can you attach all your entity definitions? Also, please paste the output of below command. Thanks! hadoop fs -ls -R /apps/falcon
... View more
01-28-2016
07:02 PM
2 Kudos
Nayan Paul: There are couple of issues in your entity xml's.
1> The granularity of date pattern in the location path should be at least that of a frequency of a feed.
2> yesterday(hours,minutes): As the name suggest EL yesterday picks up feed instances with respect to start of day yesterday. Hours and minutes are added to the 00 hours starting yesterday, Example: yesterday(24,30) will actually correspond to 00:30 am of today, for 2010-01-02T01:30Z this would mean 2010-01-02:00:30 feed. Input location path in the feed xml is /falcon/demo1/data/${YEAR}-${MONTH} but frequency is in minutes. Also if you want to process data of the month please use lastMonth or currentMonth EL expression. Please refer EL expression doc for more details. Refer this doc for entity specification details. Thanks!
... View more
01-28-2016
06:36 PM
khushi kalra:
Validity of a feed on cluster specifies duration for which this feed is valid on this cluster.
Process validity defines how long the workflow should run. It has 3 components - start time, end time and timezone. Start time and end time are timestamps defined in yyyy-MM-dd'T'HH:mm'Z' format and should always be in UTC. Timezone is used to compute the next instances starting from start time. The workflow will start at start time and end before end time specified on a given cluster.
Please refer this doc for more details. Thanks!
... View more
01-28-2016
06:32 PM
1 Kudo
Benjamin Leonhardi: You need to provide correct permissions to staging and working directories. Please use falcon tutorial for reference.
... View more
01-28-2016
01:30 AM
1 Kudo
Sushil Saxena: Regarding issue in Falcon there is FALCON-1787 jira created.
If you are running Falcon process, updating "oozie.action.sharelib.for.pig" in Oozie-site.xml will not help. Falcon generates the pig action at the run time & generates the workflow, it uses pig-action.xml defined in Falcon codebase. pig-action.xml that Falcon uses does not have hive in the share lib config "oozie.action.sharelib.for.pig".
Workflow action configuration generated by Falcon is overriding the one defined in ooze-site.xml.
Few work arounds:
1> In the process entity definition add new custom property. Resubmit the process.
<properties>
<property name="oozie.action.sharelib.for.pig" value="hive,pig,hcatalog"/>
</properties>
2> Update pig-action.xml to have hive in the share lib config and repackage falcon-oozie-adaptor-<version>.jar and replace jar at "/usr/hdp/current/falcon-server/webapp/falcon/WEB-INF/lib” and restart Falcon
3> If you have Falcon code downloaded then update pig-action.xml at oozie/src/main/resources/action/process/pig-action.xml and then build falcon and reinstall it
... View more
01-27-2016
10:23 PM
@khushi kalra: Hard to figure out what's going wrong without looking at logs. Can you tail & provide falcon.application.log when the error occurs? Also can you paste the output for hadoop fs -ls -R /apps/falcon? Thanks!
... View more
01-27-2016
07:19 PM
@Benjamin Leonhardi: Can you make sure retention coordinator is generated for the feed scheduled? Debug logs will go to falcon.applicaiton.log.
... View more