NiFi Hardware Recommendation

[#1]

I have 700+ devices that generates 4MB ~ 1GB file per device (let's say per hour) , Is there any minimum hardware specs for production?

[#2 · Accepted solution]

This is an extremely rough guess and it really depends on a lot of factors, but if you assume all devices produce 1GB an hour thats 700GB/hour, thats about 200MBs/second. You are probably looking at a 7-8 node NiFi cluster, where each node has 16-24 cpus, 16+ GB RAM, 10 Gigabit NICs. You will also need to ensure proper separation of the repositories on different disks.

[#3]

@Bryan Bende estimate, average data collected is 10~20GB per day, lets say 20GB, it is safe to say to use a single node nifi?

[#4]

Yes that is a lot less than 700 devices generating 1GB each per hour 🙂

Most people want some kind of redundancy for production so you may still want to a smaller cluster, say 3 nodes, but yes you should be able to do that on one node.

[#5]

Great! thanks @Bryan Bende!