Generate Sequence number in Apache NIFI

[#1]

How to generate a sequence number in Apache NIFI.

For example in my use case, I want to rename each file based on YYYYMMDD****, the '*' should be the sequence number for that particular day.

At the same time, I also need to keep all the YYYYMMDD=max(sequence_number) for that day in cache/filesystem for future reference.

[#2]

Hi @Arullaldivakar Jayapalan

You can do this with UpdateAttribute processor and the state feature. Configure it as below

As you can see the results has a filename attribute as you want

Does this answer your question?

[#3]

Thanks @Abdelkrim Hadjidj, It answers my question partially.

How to reset the seq number at the beginning of everyday and also we need to store all these date and max(seq) as key/value pair in some reliable filesystem for future reference.

Is it possible to achieve the same in NIFI?

Currently I have tried the same use case using DistributedMapcache ,FetchDistributedMapcache & PutDistributedMapcache, but the problem is , the updates to the cache are not atomic.

Thanks for your help.

[#4]

@Arullaldivakar Jayapalan

For 1, I think you can use Rules feature of UpdateAttribute : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/or...

You can for instance add a rule to compare between current date (now(YYYYMMDD) and the one stored in the state. If it's different, set seq to zero. You can have three attributes for instance

• Date
• Seq
• Filename which is a concat of the two

For storing (date,max), I think the best is to use an external data store (MySQL for instance) and use NiFi to store the correct values. NiFi is not meant to keep data ... it's a data mover like DHL.

Please let me know if you succeed in implementing this.

Thanks