Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to get Nifi to use a timestamp as a directory name?

How to get Nifi to use a timestamp as a directory name?

New Contributor

I have Nifi taking a number of json files from Kafka. I set up a "ConsumeKafka" to start the process. Now I need to read the json file's "creationDate" field, which is a normal UNIX timestamp (i.e., 2019-01-19T04:34:28.527722+00:00). It then needs to take part of that date and format it so it matches a corresponding HDFS directory, so the files can go into a directory that matches the date - i.e., the directories are named "01-19-2019", "01-20-2019", "01-21-2019", etc.

I was thinking of setting up an "EvaluateJsonPath" processor with a property of "creationDate" and a value of "$.creationDate". Then there would be an "UpdateAttribute" processor with a property of "creationDate" and a value of "${creationDate('yyyy/MM/dd HH:mm:ss:SSS'Z'):format('MM-dd-yyyy')}" Finally there would be a "PutHDFS" processor with a directory of "/${creationDate}".

I'm not sure about the expression (or if this is going to work the way I think it is). Especially because the directory format doesn't quite match the timestamp format. Maybe the directory names can change, but I'm going to assume that they can't. Any thoughts on how I can make this work?

Don't have an account?
Coming from Hortonworks? Activate your account here