IS THERE A PROCESSOR IN NIFI THAT CAN CREATE MANY DIRECTORY IN HDFS

[#1]

Hello everyone, I've start working with nifi last week so sorry if my question is an obvious one.

I'm currentlty taking data in a Kafka Topic, those data are in Json format. I'm scaning every Json document and depending of the value of some fields, i would like to sort them in a directory with the name of the value.

for example if i have :

{"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"nifi"}

{"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"Kafka"}

==> I would like to extract the timestamp value and convert it into "20160706" and then create a directory with it.

After that every Json with the same timestamp will be route there. Is it possible ? or I'm I dreaming ?

I initially was doing this with spark but i think it's a waste to use spark just for this. If my question wasn't clear enought i can give more details.

Thanks in advance

[#2 · Accepted solution]

See this post: https://community.hortonworks.com/questions/21011/how-i-extract-attribute-from-json-file-using-nifi....

And some more information on creating directories:

https://kisstechdocs.wordpress.com/2015/01/18/simple-tasks-in-nifi-file-objects-by-date/

Look at the split route example here: https://community.hortonworks.com/articles/961/a-collection-of-nifi-examples.html

Also useful for routing and scripting based on values: https://blogs.apache.org/nifi/entry/indexing_tweets_with_nifi_and

[#3]

Thank you very much for your Help sir and have a good day or night depend on where you are 🙂