Hi Team, Currently i am using 5 node cluster of NIFI - 1.27 version. node have 47 Gb RAM My NIFI heap memory level keep on increasing and crashing by reaching 150 hours. Frequency of data is around 100 Gb in a day. I am using variety of processors (Kafka, split and merge , distributed cache processors,Attribute level processors, hive3streaming processor,(select and put)Execute SQL (postgre,SSIS),Kudu processor etc). Please find nifi properties below: # JVM memory settings java.arg.2=-Xms16g java.arg.3=-Xmx20g java.arg.7=-XX:ReservedCodeCacheSize=512m java.arg.9=-XX:+UseCodeCacheFlushing java.arg.23=-XX:ParallelGCThreads=8 java.arg.24=-XX:ConcGCThreads=4 java.arg.25=-XX:G1ReservePercent=10 java.arg.26=-XX:+UseStringDeduplication java.arg.27=-XX:InitiatingHeapOccupancyPercent=25 java.arg.28=-XX:MaxGCPauseMillis=200 java.arg.40=-XX:SurvivorRatio=8 java.arg.41=-XX:NewRatio=3 java.arg.42=-Xmn6g java.arg.13=-XX:+UseG1GC nifi.content.repository.archive.max.retention.period=7 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.provenance.repository.max.storage.time=30 hours nifi.provenance.repository.max.storage.size=10 GB nifi.provenance.repository.rollover.time=10 mins nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.queue.swap.threshold=10000 Tagging @SAMSAL @MattWho for quick response Thanks, Krish
... View more
