How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

[#1]

I'm trying to set Falcon retention policy for a set of folders with unique naming convention. Almost all the demos and references documentation use simple folder names like "${YEAR}-${MONTH}-${DAY}" and I cant seem to figure out how to make this work. The folders are named "startday=2014-10-01", the later part is the year,month,date.

Basically there's a fixed string in front of the date, and I'd like to be able to mention in the location tag. I tried the following but it didnt work.

<location type="data" path="/user/falcon/retentiondata/#startday=${YEAR}-${MONTH}-${DAY}"/>

[#2 · Accepted solution]

First, get rid of the hashtag in your path "#startday," assuming that's not a typo. The folder name examples you're referring to are actually showing sample token replacement patterns.

For example, this:

<location type="data" path="/user/falcon/retentiondata/startday=${YEAR}-${MONTH}-${DAY}"/>

will resolve to something like this:

/user/falcon/retentiondata/startday=2015-10-27

for a daily feed that begins on 10/27 and runs. The next day's "instance" (using Falcon terms) would resolve to:

/user/falcon/retentiondata/startday=2015-10-28

[#3]

Thank you @mmiklavcic@hortonworks.com, I'll test it out !

[#4]

We actually do this as well. Here's an example for retention/replication on a folder that looks like this. This resolves, much like above, to a folder called /hdfs/path/to/data/in/table/file_dt=2015-12-09. Hope this helps!

<location type="data" path="/hdfs/path/to/data/in/table/file_dt=${YEAR}-${MONTH}-${DAY}"/>