question Re: IS THERE A PROCESSOR IN NIFI THAT CAN CREATE MANY DIRECTORY IN HDFS in Archives of Support Questions (Read Only)

IS THERE A PROCESSOR IN NIFI THAT CAN CREATE MANY DIRECTORY IN HDFS

raobelina_toky_ — Fri, 08 Jul 2016 21:33:53 GMT

Hello everyone, I've start working with nifi last week so sorry if my question is an obvious one.

I'm currentlty taking data in a Kafka Topic, those data are in Json format. I'm scaning every Json document and depending of the value of some fields, i would like to sort them in a directory with the name of the value.

for example if i have :

{"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"nifi"}

{"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"Kafka"}

==> I would like to extract the timestamp value and convert it into "20160706" and then create a directory with it.

After that every Json with the same timestamp will be route there. Is it possible ? or I'm I dreaming ?

I initially was doing this with spark but i think it's a waste to use spark just for this. If my question wasn't clear enought i can give more details.

Thanks in advance

Re: IS THERE A PROCESSOR IN NIFI THAT CAN CREATE MANY DIRECTORY IN HDFS

TimothySpann — Fri, 08 Jul 2016 21:58:37 GMT

See this post: https://community.hortonworks.com/questions/21011/how-i-extract-attribute-from-json-file-using-nifi.html

And some more information on creating directories:

https://kisstechdocs.wordpress.com/2015/01/18/simple-tasks-in-nifi-file-objects-by-date/

Look at the split route example here: https://community.hortonworks.com/articles/961/a-collection-of-nifi-examples.html

Also useful for routing and scripting based on values: https://blogs.apache.org/nifi/entry/indexing_tweets_with_nifi_and

Re: IS THERE A PROCESSOR IN NIFI THAT CAN CREATE MANY DIRECTORY IN HDFS

raobelina_toky_ — Fri, 08 Jul 2016 22:58:10 GMT

Thank you very much for your Help sir and have a good day or night depend on where you are 🙂