Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi and MiNiFi s2s Scalability

Highlighted

NiFi and MiNiFi s2s Scalability

New Contributor

I have a 4 node NiFi cluster and as I scale the number of MiNiFi agents attempting to connect via s2s I notice the cluster can't keep up with the number of HTTP requests.


Specs:

  • 4 Node NiFi Cluster: 400 Jetty threads
  • Node specs 8 core / 12 GB mem
  • 2000 MiNiFi Agents: default RPG settings and about 200 Flowfiles per minute.
  • RAW s2s protocol
  • RPG configured with a list of all 4 nodes in the cluster

Behavior:

I've noticed that when the agents begin making the initial request for "/nifi-api/site-to-site" the cluster gets over loaded and stops responding. The nifi-app.log shows this message on every node:


o.a.n.c.c.h.r.ThreadPoolRequestReplicator Cannot replicate request GET /nifi-api/site-to-site because there are 100 outstanding HTTP Requests already. Request Counts Per URI = {/nifi-api/site-to-site=100}


Where there can be anywhere from 100 to 203 outstanding requests.


Question:

It seems strange to me that 4 of these servers can't handle 2000 requests to that endpoint at once. Are the agents making a request to the /nifi-api/site-to-site endpoint per each flowfile in the queue? Is there a way to cache the list of peers return on that initial api call for a longer period of time to reduce requests? Should I scale number of Jetty threads to handle more api requests?


Please let me know if I need to provide more details.