Syslog messages start with a priority which is enclosed in < > so it should have started with something like "<10> Mar 21...." https://tools.ietf.org/html/rfc5424#section-6.2.1 https://www.ietf.org/rfc/rfc3164 (section 4.1.1) Regarding using ExtractText... yes if you got the date out of the content and into an attribute then you should be able to use UpdateAttribute with expression language functions to parse the date into the directory and filename you want. ... View more

Not exactly the solution but led me to the solution. See my comment above. Thanks for following through on the filesize HDFS corruption due to partially appended files issue. I appreciate it. ... View more

Hi Eric, did you solved the issue? I'm having the same problem when trying to use processors ConsumeKafka and ConsumeKafka_0_10 to read messages from kafka 0.10.1 (HDP 2.6 without kerberos). I did tests with different parameter settings and the processors starts without error but never receives the messages. I had same results executing the kafka-console-consumer shell with bootstrap-server parameter. Can you help me? Any idea of what's its going on? ... View more

I accepted this because your solution was in a comment down below, for future reference for others. Using the colon in the filename was the problem. ... View more

I know this was from awhile ago but I just noticed with this solution, the appended data starts right at the end of the previous event causing data loss cause two events get smooshed together. Do you have this issue when you do this and how do you fix it? Example: Apr 11 05:00:39 fw01 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.11.x.x from 00:xx:xx:xx:xx:cd to 00:xx:xx:xx:xx:a3Apr 11 05:00:39 fw01 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.11.x.x from 00:xx:xx:xx:xx:a3 to 00:xx:xx:xx:xx:cd ... View more

@Eric Lloyd With the above configuration, it would only take 1 FlowFile to be assigned to a bin before that bin was marked eligible for merging. There is nothing there that force the processor to wait for other FlowFiles to be allocated to a bin before merge, Both minimums are set to 1 FlowFile and 0 Bytes. In order to actually get 100,000 Flowfiles (this is high and may trigger OOM), there would need to be 100,000 Flowfiles all with the same correlation attribute value in the incoming connection queue at the time the processor runs. This is almost certainly not going to be the case. The Max bin age simply sets an exist strategy here. It will merge a bin regardless if minimums have been met if the bin age has reached this value. You may want to set more reasonable values for your mins and also consider using multiple mergeContent processors in series to step up to the final merged number you are looking for. Thanks, Matt ... View more

This post describes the behavior well: https://stackoverflow.com/questions/32390265/what-determines-kafka-consumer-offset ... View more

Perfect. Thank you. Still learning how to rethink data ingestion from Flume to Nifi. ... View more

That worked! Thank you!! ... View more

Thanks I gathered that. So it requires two clusters to have both HDP and HDF. Unfortunate. Im still struggling to understand why Kafka and Storm is on both and not Nifi... ... View more