Adding timestamp to flow file content

[#1]

Hi all,

I've currently got a simple processor set up to consume data from a Kafka topic and write it unmodified to hdfs.

The way it works right now is that one message makes up one line of text in a file. I want to append, to each message, a timestamp of when the message is processed in nifi before running the PutHDFS processor. Allowing me to have on hand, in hdfs, the date and time it was processed by nifi. I'm unsure of the best way to do it. I've currently got a MergeContent processor set up and I think I could do some funky stuff with the demarcator in there but it feels like there should be a more elegant way to do so.

Thanks in advance, any help would be appreciated.

[#2 · Accepted solution]

You want to use the ReplaceText processor [1] on each of the messages after they are consumed from Kafka. Set the "Replacement Strategy" to "Append" and the dynamic property to use Expression Language[2] to set the timestamp (ie. "${now()}").

Then merge the messages before sending the HDFS like normal.

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ReplaceText/in...

[2] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html

[#3]

Thanks a lot. I never noticed the "Prepend" setting in ReplaceText before but this worked a treat! Appreciate it!

[#4]

Hi @Sean Byrne,

If you want to update the filename before updating the file in HDFS, I would suggest using UpdateAttribute to update attribute 'filename' using expression language.

If you want to insert the date inside the flow file content, you could use ReplaceText processor which allows you to leverage the expression language and insert the date where you want and with the format you want.

Hope this helps.

[#5]

Thanks Pierre, I'll try out the ReplaceText processor and let you know. Appreciate the quick reply!