Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi partition file by date

avatar
New Contributor

I'm trying to split a FlowFile into multiple different files by date.

So imagine that you are receiving logs and you want to save as a Hive partitioned table so for example all records with date 2016-01-01 into directory dt=2016-01-01.

1 ACCEPTED SOLUTION

avatar
Super Mentor
@Santiago Ciciliani

Do you have any idea how many log lines per FlowFile?

A suggested dataflow may look like this:

9069-screen-shot-2016-11-03-at-84646-am.png

The SplitText processor is used to breakup your incoming log files in to many smaller FlowFiles that can more easily be handled by the RouteText processor without running out of heap memory. This is done by setting the line split count property. Depending on how much heap you have configured for your NiFi and size size of each log line really determines how many logs line you can have per split FlowFile.

The RouteText processor evaluates the entire FlowFiles content and routes groups of logs lines to a "dt" relationship:

9070-screen-shot-2016-11-03-at-85139-am.png

The UpdateAttribute processor (Optional) will create a "dt" attribute from the "RouteText.Group" attribute. YOu can use thsi attribute later to define the Hive partition table:

9091-screen-shot-2016-11-03-at-85318-am.png

The MergeContent processor (Optional) is used to combine FlowFiles with matching values (dates) in the "RouteText.Group" attribute back in to a single FlowFile.

9092-screen-shot-2016-11-03-at-85730-am.png

Don't forget to set the number of entries and max bin age properties to maximize this processors usage.

Route the "Merged" relationship from this processor to your Hive based processor.

Thanks,

Matt

View solution in original post

4 REPLIES 4

avatar
Expert Contributor

If the date is an attribute named 'dt' you can use an attribute variable in the directory path using the syntax ${dt} .

avatar
Super Mentor
@Santiago Ciciliani

Do you have any idea how many log lines per FlowFile?

A suggested dataflow may look like this:

9069-screen-shot-2016-11-03-at-84646-am.png

The SplitText processor is used to breakup your incoming log files in to many smaller FlowFiles that can more easily be handled by the RouteText processor without running out of heap memory. This is done by setting the line split count property. Depending on how much heap you have configured for your NiFi and size size of each log line really determines how many logs line you can have per split FlowFile.

The RouteText processor evaluates the entire FlowFiles content and routes groups of logs lines to a "dt" relationship:

9070-screen-shot-2016-11-03-at-85139-am.png

The UpdateAttribute processor (Optional) will create a "dt" attribute from the "RouteText.Group" attribute. YOu can use thsi attribute later to define the Hive partition table:

9091-screen-shot-2016-11-03-at-85318-am.png

The MergeContent processor (Optional) is used to combine FlowFiles with matching values (dates) in the "RouteText.Group" attribute back in to a single FlowFile.

9092-screen-shot-2016-11-03-at-85730-am.png

Don't forget to set the number of entries and max bin age properties to maximize this processors usage.

Route the "Merged" relationship from this processor to your Hive based processor.

Thanks,

Matt

avatar
Master Guru

From the NIFI User Group Mailing List by @jwitt:

Split with Grouping:

Take a look at RouteText. This allows you to efficiently split up

line oriented data into groups based on matching values rather than

spilt text which will be a line for line split.

Merge Grouped Data:

MergeContent processor will do the trick and you can use correlation

feature to align only those which are from the same group/pattern.

Write to destination:

You can write directly to HDFS using PutHDFS or you can prepare the

data and write to Hive.

avatar
New Contributor

Hello, thanks everyone for the prompt response.

With some aid I was able to figure it out

Mostly my problem was to understand the difference between the Grouping Regular Expression and extracting the date parameter which in my case are pretty much the same expression.

Also I have to admit that the RouteText.Group attribute was not something easy to find even in the documentation.

I feel that reading a TCP connection with logs and store it partitioned directly to a Hive table should be a fairly common use case, so I'm attaching the template as a grain of sand contribution.

recordtexttopartition.xml

Thanks again