Hi All, Thanks a lot for the help. I am stuck in a situation, I have lists3 and fetchs3 processors. fetchs3 brings the files from s3, there is a querydatabasetable processor which queries the table and brings in the list of filename already written in DB. Now we need to compare the filenames from fetchs3 with the filenames from querydatabasetable and only write those files which have not been written to table before. My approach was using using detectduplicate processor, writing the result of querydatabasetable to detectduplicate and then compare it to fetchs3. But I am not able to achieve it in NiFi, any suggestions ideas will help Thanks Dhieru ... View more

Hi All, I installed winlogbeat on windows, and wanted to ship logs using ListenBeats in nifi. However when I check output of winlogbeat there is nothing to stream to tcp or udp.Only options are Elasticsearch ,Logstash, Kafka, Redis, File, Console, Cloud. Any way to stream to tcp port so that listenbeats can listen on the port. Thanks in advance dhieru ... View more

Hi All, Thanks a lot to this awesome community. My use case is like as follows We use listentcp porcessor to listen to small firewall events max size 200 bytes. But we batch them at Listentcp level to increase network I/O and performance. Then we split them into individual flow files and do some processing and enrichment and publish them to Kafka. However, our kafka is not able to keep up with rate of flow. We have millions of flowfiles queued up. I read that kafka can publish millions of small message per second. What all properties do I need o configure in publish kafka so to increas its performane. I tried tuning max.request.size however it is the maximum size of each message so no help I tried adding one more property buffer.memory to buffer small message together and then publish still no help should I also add one more proerty called batch.size (controls how many bytes of data to collect before sending messages to the Kafka broker. Set this as high as possible, without exceeding available memory. The default value is 16384.) and linger.ms (linger.ms sets the maximum time to buffer data in asynchronous mode. For example, a setting of 100 batches 100ms of messages to send at once. This improves throughput, but the buffering adds message delivery latency.) In my opinion I should buffer for all the small messages atleast (100000 messages ) then write to kafka topic, this will increase the netwrok I/o as well less writes. I just not sure which properties will help me here. Thanks Dhieru ... View more