We have a CM instance that's currently administering 1,340 nodes. From a prior discussion with another team at Cloudera, it came out that CM is only meant to administer 1,000 nodes in it's current form. Due to that, we're looking at splitting out our clusters to another CM instance, but that's the long term plan. For the short term, CM has become very sluggish.
A few examples is that, now, if you go to the hosts page in CM, to show all hosts, chrome will say that the tab becomes unresponsive about 2/3 of the time, just because the tab almost never loads. We're unable to go back much more than 2 pages in the command history, and whenever you give a command, such as cycle service, we typically see a 2-4 second delay.
We've already done some tuning on our instances. We started with 8gb memory and 2 cores. We've sized up to 30gb memory and 6 cores. We also initially had 4 vm's, with all vm's running multiple services. We added 2 more vm's so we could isolate the host monitor and service monitor, since these two services seemed to be the heaviest used services.
We're currently sitting at 18gb xms and xmx options on our main cm server service, with 3gb xmn. Our other services typically sit between 8-12gb heap.
What other tuning options are recommended to improve performance?
