Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Nifi-Kafka hardware recommendation and architecture

New Contributor

Hello all,

We are working on a design for a HDF/HDP solution. The initial setup would be:

  • 3 physical nodes for Kafka (300MBpseg), each with 24 cores-128GB RAM-48TB local disk (that would yield to 225MBpseg initially if not mistaken)
  • 3 physical nodes for storage/worker, each with 28 cores-256GB-114TB local disk
  • nodes for management (3 in total)

Nifi is also required, but we have doubts about the mix of Nifi and Kafka servers. Would it be possible to run both the Kafka and Nifi services in the same physical servers already provisioned for Kafka or should we provision additional physical servers for Nifi?

In case that the services would be able to me mixed, there will be enough horsepower to handle the expected 300MBpseg?

We notice that the hardware requirements for both solutions are quite different,

Thanks,

1 REPLY 1

You should provision additional servers strictly for NiFi paying careful attention to recommendations for disk allocation and partitioning specifically for NiFi and its necessary repositories.

I would only recommend mixing services in dev and test environments.