This article aims to show how to planning a Nifi Cluster following the best practices.
1) Hardware Provisioning
2) Hardware Considerations for HDF
- General Hardware
A key design point of NiFi is to use typical enterprise class application servers.
- Machine Class
A NiFi cluster consists of a single class of machine
Balanced NiFi Node:
In-rack backplane/Top-of-rack Switch:
NIFI is designed to
3) HDF Disk Partition Baseline
4) Disk Partitioning – Nifi Nodes (Repositories)
Default Cluster Recommendation
not provided with information to gauge the rate and complexity of data flow,
start with a default cluster of three nodes. Three nodes are needed for HA by
Zookeeper Quorum process.
SKU is priced for cores, but it can be split up. So, a 16 core SKU can be split
into 3 machines of 4 cores each. More cores per node will improve throughput
(up to an extent).
for, say, 50MB/s sustained throughput for average Flow is:
6) NiFi Clusters Scale Linearly
Have you checked Nifi throughput using Content Repo in a JBOD mode instead of Raid? Basically, let application decide for the distribution of data.