Director Not Increasing Heap of Host Monitor

Question

Just wanted to bring up a concern about Director that it doesn't increase the non-java memory allocated to the host monitor (or at least not enough) when deploying a large amount of hosts to a deployment which will cause the host monitor to fail to start (due to CM specifying an error condition in 5.16.2). This doesn't fail the bootstrap (thankfully) but does require the user to go into the CM and manually set the recommendations for the Host Monitor before being able to start it and get metrics about the hosts.

jadair · Accepted Answer

For every choice of semantics for these properties, we found failure scenarios that would result in good values overwriting bad values, either in CM during bootstrap or grow, or in Director's deployment or cluster template during refresh. The blacklist was the least-bad solution.

The blacklisting will not result in errors if you provide the properties--they will just be silently ignored when Director passes the configuration to CM. With the blacklists in place, CM becomes the source of truth for these properties, so you will still be able to set them in CM if the autoconfigured values are not acceptable. If for some reason you have problems bootstrapping without them, the bootstrap and update blacklists are separately configurable, so you might be able to provide them at bootstrap. I can't recall what the potential failure scenarios were there.

Da_01 · Answer

For reference, what I'm doing to address this is setting the following in the api call to create the deployment in DeploymentTemplate(configs=...):

"SERVICEMONITOR" : { "firehose_non_java_memory_bytes" : "6442450944" }, "HOSTMONITOR" : { "firehose_non_java_memory_bytes" : "6442450944" }

Which makes sure the host monitor and smon have 6 GB configured for the non java memory.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.