Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi Hardware Recommendation

Solved Go to solution

NiFi Hardware Recommendation

Contributor

I have 700+ devices that generates 4MB ~ 1GB file per device (let's say per hour) , Is there any minimum hardware specs for production?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi Hardware Recommendation

This is an extremely rough guess and it really depends on a lot of factors, but if you assume all devices produce 1GB an hour thats 700GB/hour, thats about 200MBs/second. You are probably looking at a 7-8 node NiFi cluster, where each node has 16-24 cpus, 16+ GB RAM, 10 Gigabit NICs. You will also need to ensure proper separation of the repositories on different disks.

View solution in original post

4 REPLIES 4
Highlighted

Re: NiFi Hardware Recommendation

This is an extremely rough guess and it really depends on a lot of factors, but if you assume all devices produce 1GB an hour thats 700GB/hour, thats about 200MBs/second. You are probably looking at a 7-8 node NiFi cluster, where each node has 16-24 cpus, 16+ GB RAM, 10 Gigabit NICs. You will also need to ensure proper separation of the repositories on different disks.

View solution in original post

Highlighted

Re: NiFi Hardware Recommendation

Contributor

@Bryan Bende estimate, average data collected is 10~20GB per day, lets say 20GB, it is safe to say to use a single node nifi?

Highlighted

Re: NiFi Hardware Recommendation

Yes that is a lot less than 700 devices generating 1GB each per hour :)

Most people want some kind of redundancy for production so you may still want to a smaller cluster, say 3 nodes, but yes you should be able to do that on one node.

Highlighted

Re: NiFi Hardware Recommendation

Contributor

Great! thanks @Bryan Bende!

Don't have an account?
Coming from Hortonworks? Activate your account here