Support Questions
Find answers, ask questions, and share your expertise

(Small demo environment) Any tips for distributing system memory for Cloudera Manager Services? What about Kafka I/O on virtual environment?


I'm working on a small demo environment that is currently utilizing:

1x 3 broker Kafka cluster
1x zookeeper host
1x SCM host

1x postgres db


SCM is reporting that I need to increase system memory for my SCM host pretty much every time I add any service (started at 2gb, upgraded to 16, now it wants 32).

I want to keep things optimized, but I feel like maybe I'm not setting things up properly. I'm running all 4 manager services on the single SCM host. I understand that Kafka is not recommended on VM environments due to I/O issues. I would like to understand how to make the most of this (would 1 kafka node be preferential to 3 in this case?)

How should I distribute memory allocation on my SCM host for the most performance? I would like to keep the SCM host at 16gb RAM if possible. This demo environment is not going to be dealing with huge amounts of data, and is not even going to be Highly Available. Where should I cut corners to make the most of my available resources?