Re: Director Not Increasing Heap of Host Monitor

jadair · ‎07-19-2019

For every choice of semantics for these properties, we found failure scenarios that would result in good values overwriting bad values, either in CM during bootstrap or grow, or in Director's deployment or cluster template during refresh. The blacklist was the least-bad solution. The blacklisting will not result in errors if you provide the properties--they will just be silently ignored when Director passes the configuration to CM. With the blacklists in place, CM becomes the source of truth for these properties, so you will still be able to set them in CM if the autoconfigured values are not acceptable. If for some reason you have problems bootstrapping without them, the bootstrap and update blacklists are separately configurable, so you might be able to provide them at bootstrap. I can't recall what the potential failure scenarios were there.

jadair · ‎07-19-2019

Can you please clarify whether you were already using those custom settings and they are not being applied correctly, or whether you started providing those custom settings as a solution for the problem you raised in this post? If it's the latter, I'm glad you found the custom settings. That was going to be my recommendation. What version of Director are you on? While implementing improvements to the deployment/cluster refresh process in 6.2, we ran into some bugs around those settings getting overwritten during refresh. There is a new blacklist in place to avoid overwriting those properties (so that CMs autoconfigured values don't get smashed), but that could prevent your custom values from taking effect. So if you run into any problems on 6.2 or later, please let us know.

jadair · ‎07-19-2019

According to the comments on the Jira, the properties will not include hikari. That is consistent with the recommendation in the previously linked stackoverflow thread.

jadair · ‎07-19-2019

@Da It turns out that the inability to configure these properties was a Director bug which has been fixed for our upcoming 6.3 release. Once you are on 6.3, you will be able to configure the setting (for the Hikari connection pool, as it turns out) using the spring.datasource.maximumPoolSize property. I am sorry that there is no workaround for earlier releases.

jadair · ‎07-19-2019

We're relying on springboot, and are not doing anything special. I notice that in the application.properties embedded in the jar, some of the documented tomcat properties, like testOnBorrow and validationQuery, are being set with a prefix of spring.datasource.tomcat. Based on that stackoverflow answer, I wouldn't expect that to be required for maxActive, but give it a try and let me know if it works.

jadair · ‎07-19-2019

I can't confirm that your problem is caused by the connection pool size, and have never adjusted the connection pool settings manually, but Director uses spring-boot so you should be able to set the appropriate properties in application.properties as described in this stackoverflow question: https://stackoverflow.com/questions/25573034/spring-boot-how-do-i-set-jdbc-pool-properties-like-maximum-number-of-connection The property prefix should be spring.datasource, and the supported properties should be these: http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes For the pool size specifically, you should be able to set it like this: spring.datasource.maxActive=200

jadair · ‎07-18-2019

It would be helpful for us to see the logs. There should be additional info logging for the individual connection attempts ("Attempting connection to <endpoint>").

jadair · ‎07-15-2019

It looks like we failed to update the documentation when we added support for this in 6.2.0. Good catch! The cluster refresher should detect changes to the instance type made in the Azure portal and update the cluster template with the new information. If this results in an instance group being inconsistent (with some instances on the old instance type and some on the new), grow operations will be blocked for the cluster. If the instance group is made consistent (by either changing all the instances to the new instance type, or by shrinking away instances using the old instance type), grow will be enabled, and a subsequent grow operation will use the new instance type.

jadair · ‎04-04-2019

Support for CentOS 7.6 was added in Cloudera Altus Director 6.2, which released today. Altus Director does not validate the CentOS version, but CentOS 7.6 has not been tested with older Altus Director versions.

jadair · ‎07-26-2018

Your prior information was clear. The mismatch was in my terminology. The term "deployment" is used in the APIs, while the UI says "Cloudera Manager". These are synonymous. Sorry for the confusion! Please see if you can terminate one of the failed Cloudera Managers, rather than one of the corresponding clusters. If that fails, I'm sorry to say that there is currently no safe way to force deletion of the deployment or cluster from Director's database. If this is purely a test setup, you could delete any successful deployments, stop the Director service, then start from scratch (on H2, by deleting the director database file from disk, or on MySQL, by creating a new empty database for Director with the appropriate permissions). When you start Director again, it will rebuild the database contents. If this is not feasible, you can just ignore the failed deployments and clusters.

jadair · ‎07-26-2018

There is a technical distinction in Cloudera Director between a deployment and a cluster. Each Cloudera Director instance can manage multiple environments (each corresponding to an account on a cloud provider), each of which can have multiple deployments (each corresponding to a single Cloudera Manager instance), each of which can have multiple clusters. I am specifically asking if you have tried terminating the deployment, rather than the cluster. In your last answer you still mentioned terminating the cluster.

jadair · ‎07-26-2018

Are there currently orphaned resources (such as instances) in Azure? If so, can you try deleting them manually and then trying to terminate the deployment via the Cloudera Director UI?

jadair · ‎07-26-2018

You mention that the deployment failed, and that you are trying to terminate the cluster. Have you tried terminating the deployment directly in the Cloudera Director UI?

jadair · ‎07-26-2018

Glad you got past this problem!

jadair · ‎07-25-2018

The Cloudera Manager password can be updated in Cloudera Director in a variety of ways, which are not currently well-documented in the Cloudera Director User Guide. If you are using the Cloudera Director UI, you can use the "Update Cloudera Manager Credentials" option in the dropdown on the deployment page. A deployment update (PUT) through the API (using the API console or Java/Python SDKs) can also do it, since username and password are among the legal template fields to change.

jadair · ‎07-24-2018

You can safely ignore the "missing endpoint address string" warning. Do you have a stack trace or additional error message about the failure to reach Cloudera Manager? Cloudera Director must have SSH access to the CM instance and each of the cluster instances. You can achieve this by running Cloudera Director in the same subnet as the instances, by configuring some sort of VPN, or by configuring the instances to have public IP addresses.

jadair · ‎06-21-2018

Glad you figured this out. Thanks for posting your solution for other forum users.

jadair · ‎06-20-2018

I'm glad you figured this out. There are some other places you can look for information about provider-specific configuration properties as well. There are reference configuration files here: https://github.com/cloudera/director-scripts/tree/master/configs From the help menu in the Director UI, you can access the API console, and the provider-metadata endpoints there will show you all the valid provider-specific configuration properties. If you're comfortable reading code, the plugins that enable Director to work with multiple cloud providers are open source. If you search those repositories for ConfigurationPropertyToken, you will see the classes that define the metadata for the provider-specific configuration properties. The AWS repository, for example, is here: https://github.com/cloudera/director-aws-plugin/tree/v2

jadair · ‎04-09-2018

Glad your problem is solved. Thanks for letting the community know the solution. Hopefully if someone else ends up in your situation they will find your answer helpful.

jadair · ‎11-13-2017

Director's deployment and cluster refreshers should detect this sort of thing and update the deployment and cluster representations (respectively) in the database with the updated version information. Can you please check the server application.log to see if either of them encountered problems?

jadair · ‎11-13-2017

Yes, this is most likely a consequence of the other issue. We should try to address that one first.

jadair · ‎11-03-2017

The root cause should be a little earlier in the log. The log snippet you provided starts when Director is trying to clean up resources in response to the error.

jadair · ‎10-04-2017

Yes. Are you attempting to specify a custom Name tag in your instance template? Cloudera Director by default uses that tag to attach the name that it generates for the instance. If it is not critical for you to provide a value for the Name tag, you should remove it from your configuration. If it is critical, it is possible to configure Director to use custom tag names. See the "Configuring Cloudera Director to Use Custom Tag Names on AWS" section of the Cloudera Director User Guide. There was a bug around this functionality in Director 2.5.0, but it should work correctly in Director 2.5.1 and later.

jadair · ‎10-04-2017

I'm sorry you're having trouble getting your cluster running. You are trying to create the cluster through the Cloudera Director UI, right? If so (or if you're using the bootstrap-remote command from the command line), you should look in Cloudera Director server's application.log for an earlier exception giving the specific failure during instance allocation.

jadair · ‎10-03-2017

It turns out that the agent installation log is not currently part of the diagnostic data that Director tries to download from CM. If you do not get a useful error from the Director server log, you should SSH into the CM instance and search the /tmp directory for scm_prepare_node.log. There may be more than one of these because of the retries. Those log files may contain additional information about the reason for failure.

jadair · ‎10-03-2017

Try /var/log/cloudera-director-server/application.log. If it's not there use "sudo find . -name application.log" from some suitably high-level directory.

jadair · ‎10-03-2017

Have you already checked the Director server log for more details about the failure? In addition to looking for ERROR log messages and stack traces, you can look to see if Director was able to download diagnostic data from Cloudera Manager with additional information about the reason for agent installation failure.

jadair · ‎08-23-2017

This looks like a duplicate of your other post. Can you please confirm that this issue is resolved?

jadair · ‎08-23-2017

I'm glad your problem is resolved. Thank you for posting the details so other forum members can benefit from your experience.

jadair · ‎08-23-2017

Sorry for the delayed reply. Unfortunately, there is not enough info in that log snippet to determine why the agent installation failed. Do you know whether the agent installation failed on the CM instance or one of the cluster instances? If the former, look earlier in the Director log to see if there are any additional details about the specific failure. If the latter, also look in the CM log.

Online	Offline
Last Visited	‎08-19-2019 12:16 PM

Date Registered	‎10-28-2014 12:50 PM
Last Visited	‎08-19-2019 12:16 PM
Total Messages Posted	71
Total Kudos Received	11

Re: Director Not Increasing Heap of Host Monitor

Re: How to increase database pool for Director?

Re: How to increase database pool for Director?

Re: Azure: Changing instance type post deployment

Re: Cloudera Altus support of CDH on CentOS 7.6

Re: Director Not Increasing Heap of Host Monitor

Re: Director Not Increasing Heap of Host Monitor

Re: How to increase database pool for Director?

Re: How to increase database pool for Director?

Re: How to increase database pool for Director?

Re: How to increase database pool for Director?

Re: Altus Director 6.2: Failure to connect to Azur...

Re: Azure: Changing instance type post deployment

Re: Cloudera Altus support of CDH on CentOS 7.6

Re: Cannot terminate cluster

Re: Cannot terminate cluster

Re: Cannot terminate failed Cloudera Director depl...

Re: Cannot terminate cluster

Re: Cluster Bootstrap Fails Google Cloud

Re: Missing endpoint address string for publicIpAd...

Re: Missing endpoint address string for publicIpAd...

Re: Cluster install with Director hangs on KAFKA p...

Re: Create Environment API - config object

Re: Name node not starting after computer restarts

Re: Cloudera Director not reflecting upgraded CDH ...

Re: Attempt to add worker scaler node using Cloude...

Re: Cloudera Director failes to bootstrap Cloudera...

Re: Cloudera Director fails to bootstrap Cloudera ...

Re: Cloudera Director fails to bootstrap Cloudera ...

Re: Cloudera Director Fails on 'Deploying Cloudera...

Re: Cloudera Director Fails on 'Deploying Cloudera...

Re: Cloudera Director Fails on 'Deploying Cloudera...

Re: com.cloudera.launchpad.pipeline.UnrecoverableP...

Re: com.cloudera.launchpad.pipeline.UnrecoverableP...

Re: ClouderaDirector unable to bootstrap - Insuffi...