Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

Solved Go to solution
Highlighted

How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

New Contributor

I'm trying to set Falcon retention policy for a set of folders with unique naming convention. Almost all the demos and references documentation use simple folder names like "${YEAR}-${MONTH}-${DAY}" and I cant seem to figure out how to make this work. The folders are named "startday=2014-10-01", the later part is the year,month,date.

Basically there's a fixed string in front of the date, and I'd like to be able to mention in the location tag. I tried the following but it didnt work.

<location type="data" path="/user/falcon/retentiondata/#startday=${YEAR}-${MONTH}-${DAY}"/>

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

New Contributor

First, get rid of the hashtag in your path "#startday," assuming that's not a typo. The folder name examples you're referring to are actually showing sample token replacement patterns.

For example, this:

<location type="data" path="/user/falcon/retentiondata/startday=${YEAR}-${MONTH}-${DAY}"/>

will resolve to something like this:

/user/falcon/retentiondata/startday=2015-10-27

for a daily feed that begins on 10/27 and runs. The next day's "instance" (using Falcon terms) would resolve to:

/user/falcon/retentiondata/startday=2015-10-28

3 REPLIES 3

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

New Contributor

First, get rid of the hashtag in your path "#startday," assuming that's not a typo. The folder name examples you're referring to are actually showing sample token replacement patterns.

For example, this:

<location type="data" path="/user/falcon/retentiondata/startday=${YEAR}-${MONTH}-${DAY}"/>

will resolve to something like this:

/user/falcon/retentiondata/startday=2015-10-27

for a daily feed that begins on 10/27 and runs. The next day's "instance" (using Falcon terms) would resolve to:

/user/falcon/retentiondata/startday=2015-10-28

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

New Contributor

Thank you @mmiklavcic@hortonworks.com, I'll test it out !

Re: How to set falcon retention policy for unconventional folder name ? " startday=2014-10-01"

Contributor

We actually do this as well. Here's an example for retention/replication on a folder that looks like this. This resolves, much like above, to a folder called /hdfs/path/to/data/in/table/file_dt=2015-12-09. Hope this helps!

<location type="data" path="/hdfs/path/to/data/in/table/file_dt=${YEAR}-${MONTH}-${DAY}"/>