Member since
12-13-2016
72
Posts
7
Kudos Received
1
Solution
03-04-2020
08:30 PM
@MattWho Thanks for the reply. Sure here i have attached my existing merge_content processor configuration and single node ec2 with 8GB of RAM. can you please clarify if flowfile size gets increase even TB's of data shall i processed with same approach multiple merge_content processor or if it is good with multi-node or increase single node maximum memory availability. JVM Configuration : # JVM memory settings java.arg.2=-Xms2048m java.arg.3=-Xmx2048m
... View more
03-04-2020
01:09 AM
Hi, I'm currently using merge_content processor as merge avro files from two sources like kafka_consumer processor and fetchHDFS file. While converting avro file into one with merge content processor yesterday around 680MB trying to convert but processor drop the file and join with new files and i can't able to recover that data also, because content_repository backup i limit. Can you please help me out for this use_case processor how much size can be good or is there any setting needs to modifiy in nifi.properties.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
-
HDFS
12-15-2019
09:26 PM
Sure thanks. @MattWho. it works!
... View more
12-12-2019
09:27 PM
Hi, We are currently using merge content processor to merge kafka messages with "minimum number of entries" = 500. when 500 messages reached it will merge as single file. That use case works fine. whenever end of the day at 11:59, is there any queue pending in merge content processor of kafka messages needs to dynamically pushed before start to the new date. Kindly help me out from this kind of use case. Workflow : kafka_consumer -> merge_content_prcessor -> puthdfs @mburgess
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi