Support Questions
Find answers, ask questions, and share your expertise

Nifi to HDFS filename missing minutes

Rising Star

We have a recurring issue we haven't been able to solve.

Take a look at our flow. We're taking logfiles, extracting date and time attributes to use as the filename when writing them to HDFS

It mostly works but we keep intermittently getting filenames that are missing the minutes field.

2017_05_16_13_55_fozziesplunkr.log

2017_05_16_13_56_fozziesplunkr.log

2017_05_16_13_57_fozziesplunkr.log

2017_05_16_13__fozziesplunkr.log <---this is the file. It contains entries from minutes 55, 56 and 57. It feels like a

15450-puthdfsmergecontent.jpg

"catch-all" file.

Our setup: 6 hosts sending to 3 topics, 2 hosts per topic. It is generating at about 10k/messages per second of 1 million events. This is just our test data.

On our sending side, we are doing TailFile -> Control Rate (1 MB) -> PublishKafka (this seems to work well)

On our receiving side, there are screenshots of one of our topics with every processor and its tabs.

We used ConsumeKafka -> ExtractText -> UpdateAttribute (Regex for timestamp from log) -> MergeContent -> UpdateAttribute (Create filename) -> PutHDFS

All of these have screenshots as shown. If anyone has had this problem and has any idea on a solution, that'd be welcome. We've tried all kinds of performances tweaks without success. Nifi logs show no warnings or errors.

Thanks.

Overview of Flow & Filenames

15447-overviewfiles.jpg

Consume Kafka & Create Filename

15448-consumekafkacreatefilename.jpg

Extract Text (regex to receive from log the timestamps and put them into attributes)

15449-extracttext.jpg

PutHDFS & MergeContent

15450-puthdfsmergecontent.jpg

3 REPLIES 3

Rising Star

Sorry I was having problems uploading. It kept timing out so I guess all three ended up posting. How do I close it? The delete option is not available (and I think its because this question has a comment on it, this comment here...)

15538-screen-shot-2017-05-18-at-91405-am.png

Rising Star

I reported it to get it removed.

; ;