Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

performance slowness and thread starvation on pipeline

Highlighted

performance slowness and thread starvation on pipeline

New Contributor

I have created a pipeline having 6 process groups which consists of 95 processors .

The max thread count set by me is 4 times the no of cores (ie 16*4=64).

when i executed the process group one by one then the egress from the process group is very fast( ie more than 100) ,but when i run the whole pipeline(ie all the process groups of pipeline at the same time) the egress comes to 100 only and I have noticed there are lots of processors with no active thread .which is slowing down the performance of pipeline .

I have also set the following parameters .

1)nifi.bored.yield.duration=2 millis (but the cpu utilization never goes beyond 60%)

nifi.queue.backpressure.count=30000

nifi.queue.swap.threshold=40000 (To stop swapping)

nifi.provenance.repository.index.shard.size=3072 MB

2)I have increased the concurrent task for each process to increase the cpu utilization still it does not increased .

3)All the processors are timer driven .

4)I have increased the heap memory to 16 GB

5)java.arg.7=-XX:ReservedCodeCacheSize=1024m
java.arg.8=-XX:CodeCacheMinimumFreeSpace=256m
java.arg.9=-XX:+UseCodeCacheFlushing

java.arg.13=-XX:+UseG1GC

I am using nifi 1.8 version .

Don't have an account?
Coming from Hortonworks? Activate your account here