question Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01" in Archives of Support Questions (Read Only)

How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

asunke — Wed, 28 Oct 2015 07:05:02 GMT

I'm trying to set Falcon retention policy for a set of folders with unique naming convention. Almost all the demos and references documentation use simple folder names like "${YEAR}-${MONTH}-${DAY}" and I cant seem to figure out how to make this work. The folders are named "startday=2014-10-01", the later part is the year,month,date.

Basically there's a fixed string in front of the date, and I'd like to be able to mention in the location tag. I tried the following but it didnt work.

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

mmiklavcic — Wed, 28 Oct 2015 07:15:17 GMT

First, get rid of the hashtag in your path "#startday," assuming that's not a typo. The folder name examples you're referring to are actually showing sample token replacement patterns.

For example, this:

will resolve to something like this:

/user/falcon/retentiondata/startday=2015-10-27

for a daily feed that begins on 10/27 and runs. The next day's "instance" (using Falcon terms) would resolve to:

/user/falcon/retentiondata/startday=2015-10-28

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

asunke — Wed, 28 Oct 2015 11:31:54 GMT

Thank you @mmiklavcic@hortonworks.com, I'll test it out !

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

landon_t_robins — Wed, 09 Dec 2015 23:15:11 GMT

We actually do this as well. Here's an example for retention/replication on a folder that looks like this. This resolves, much like above, to a folder called /hdfs/path/to/data/in/table/file_dt=2015-12-09. Hope this helps!

<location type="data" path="/hdfs/path/to/data/in/table/file_dt=${YEAR}-${MONTH}-${DAY}"/>