please add the clusterType field to your simple conf. This field is used to determine the cluster topology (e.g., what services to run, where to locate roles).   The exported conf will not include configurations that are set through CM's autoconfiguration, only those that are manually set. You may not need to set those Kudu configurations unless you want to customize them to something different than what autoconfiguration would choose. ... View more

You are correct in that Director will attempt to connect via each IP in the list, preferring to connect over the private IP. You should be fine if only one of the IPs is accessible. Director should cycle through the IPs until the configured timeout, default 20 minutes after instance allocation.   Is the cluster failing to bootstrap? There are some natural places where we would expect connectivity to fail in a transient manner. E.g., when the VM is first allocated but doesn't yet respond to ssh, when rebooting the VM. If you are seeing failures that are preventing the cluster from bootstrapping (or even causing individual instances to fail) then we would be interested in seeing the log files to investigate. ... View more

The specific error you are seeing,  "The configuration does not appear to be in a supported format", is because the convert-remote command doesn't think the config is a simple-setup style config. It checks this based on the inclusion of the "clusterType" field. see https://github.com/cloudera/director-scripts/blob/master/configs/aws.simple_setup.conf#L47-L51   Can you check your simple-setup conf file for this field?    validate-remote can validate both simple-setup and standard conf files, so it will submit the request regardless of the presence or absence of the "clusterType" field. I'm a little surprised that the validate-remote command would work on a simple-setup conf file missing the "clusterType" field, but validation only checks for a subset of problems and would not guarantee that the cluster would bootstrap properly. ... View more

GaryS,   The convert-remote command is used to convert simple-setup style conf files into full conf files. e.g., files like https://github.com/cloudera/director-scripts/blob/master/configs/aws.simple_setup.conf into files like https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf   Can you tell us a little more about what you are trying to accomplish and about the conf file you are trying to convert? ... View more

"enableEnterpriseTrial" should have no effect on java installation. Also, the java rpm should be available in the CM yum repo. What version of Director are you using and what version of CM are you installing?   I do see that  https://archive.cloudera.com/cm6/6.2.0/ shows a "last modified" date of yesterday (7/9). Perhaps there was some maintenance being performed on the repo when you ran. Can you try to run again? ... View more

followup with a little more context and clarification.   I mis-spoke when I said that " the created users are not being granted the correct role to operate on the created database." In actuality, the role provided to Altus Director is not able to create a database with an owner other than itself. The database provisioning logic attempts to create a separate role and use that as the owner for each database. SUPERUSER privileges are required to create a database with a different owner . Google cloud sql does not support giving SUPERUSER privileges (see https://cloud.google.com/sql/docs/postgres/features under "Unsupported Features").   External Database Templates should work correctly if you create your own Postgres server and provide Altus Director with a SUPERUSER role. ... View more

cgiardina,   Altus Director is not able to manage (i.e., create/delete) cloudsql postgres databases due to a permissions issue. As you can see by the error messages, the created users are not being granted the correct role to operate on the created database.   You can still use postgres databases that you create and manage yourself. See the documentation and reference conf file for information on External Databases, not External Database Templates. https://www.cloudera.com/documentation/director/latest/topics/director_external_db_using.html https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf#L629-L653 https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf#L993-L1034   ... View more

I think that this log line highlights the error.   PluginExceptionCondition{type=ERROR, exceptionInfo={message=Encountered AWS exception, awsErrorCode=InvalidParameterValue, awsErrorMessage=Duplicate tag key 'Name' specified.}}   Are you trying to add a "Name" tag to your instances? Cloudera Altus Director uses the "Name" tag, but can be re-configured to use a different tag instead. See https://www.cloudera.com/documentation/director/latest/topics/director_aws_custom_tag_names.html for instructions on how to do so.   ... View more

ppiper,   The port forwarding errors can generally be ignored. Cloudera Director is polling the ssh port on the instance to wait until the instance starts responding.   The failure is likely indicated by this in the second set of logs you included PluginExceptionCondition{type=ERROR, exceptionInfo={message=403 Forbidden { "code" : 403, "errors" : [ { "domain" : "usageLimits", "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1.", "reason" : "quotaExceeded" } ], "message" : "Quota 'CPUS' exceeded. Limit: 8.0 in region us-east1." }}} It looks like you exceeded your quota in Google Cloud.   David ... View more

robinwood,   I'm afraid that there's not enough information in your log snippet for me to tell what failed. Can you supply a larger portion of the logs preceeding this error?   Log lines marked with [ p-dfc30dc4e9a3-DefaultBootstrapDeploymentJob] are for this thread that experienced the error. Also look for some [io-thread-?] log lines that are the output from the script execution. Those may give us some more clues as to why the error occurred.   ... View more

Cloudera Director can install Kudu using conf files and the CLI bootstrap-remote command.  In addition to the linked documentation above, also see the aws.reference.conf. The cluster defined in the reference conf file includes Kudu.   If you want to install Kudu on a Director-managed cluster that already exists, you need to follow the Cloudera Manager's documentation to do so. Director will "refresh" it's models of the cluster to include any added services or configurations. See Director's documentation on  cluster management for a description of which tasks should be performed in Director vs. CM. ... View more

sg321,   Please look at this article. http://community.cloudera.com/t5/CDH-Manual-Installation/How-to-resolve-quot-Permission-denied-quot-errors-in-CDH/ta-p/36141   I find it strange that you are getting this error on a clean cluster bootstrap. Can you share your conf file? Please redact any secrets (e.g., aws keys). What version of Director are you using?   David ... View more

sg321,   You do need to look in CM to find the root cause of the First Run failure.  The CM UI has a tab "All Recent Commands". https://www.cloudera.com/documentation/enterprise/5-12-x/topics/cm_dg_view_running_recent_commands.html   First Run is composed of a series of steps. You can determine which step failed through the UI.   You can also find this information in the scm-server-log. Information about the step that failed should appear somewhere above the scm-server-log message that you posted.   David ... View more

alexmc6,   Based on the terminology used, I'm assuming that you are referring to AWS. Please refer to the Cloudera Enterprise Reference Architecture for AWS Deployments. "Appendix A: Spanning AWS Availability Zones" will provide the guidance you are looking for.   In Director, each Environment corresponds to a single region, but you can (indirectly) select the AZ used for each node by setting the subnet in the InstanceTemplate. Refer to the aws.reference.conf for an example of where you can override the subnetId and set the rackId. Note that, since a single InstanceTemplate is used for each instance group, you will need to make separate instance groups for each AZ.   David ... View more

Sentara,   Thank you very much for posting your approach here. I'm sure that it will help future community users.   David   ... View more

Sentara,   You are correct that your DNS must be able to resolve the ip-X-X-X-X@ourcompany.com names or whatever names you want your hosts to use. Definitely review AWS's documentation https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html   Here's some suggestions:   1. You can run a bootstrap script to your InstanceTemplates to add your hostname to DNS using dhclient or NetworkManager if your DNS server is configured to allow updates from your VPC instances. We have published some scripts to help people using Cloudera Director in Azure. You can refer to and adapt these scripts to your network. See https://github.com/cloudera/director-scripts/tree/master/azure-dns-scripts   2. You can brute force the problem by pre-registering all potential hostnames in your VPC or subnet with your DNS server. The subnets are configured to use IP addresses in a specified range and the hostnames are derived from those IP addresses in a known, systematic way. Thus, you can produce the expected hostnames for each IP address in your subnet and add those to your zone files.   3. You can investigate using a bootstrap script to obtain a company-specific hostname that is resolvable in your company DNS and configure your instance to use that hostname instead of the ip-X-X-X-X hostname.   Please report back any progress or solutions that work for you to share the knowledge with the rest of the community.   If anyone else in the community has any suggestions, please chime in!     Thanks, David ... View more

Always use the configuration API name with Cloudera Director. See https://www.cloudera.com/documentation/enterprise/properties/5-14-x/topics/cm_props_cdh5140_hue.html#concept_5.14.x_loadbalancer_props__section_ports_and_addresses_props for the API names.   This is a Role config (not service-wide) so the config must be placed in each instance group that contains a hue load balancer role.   e.g.,  cluster { configs { # service-wide configs are placed at the cluster level } instance-group-name { roles { HUE: [HUE_LOAD_BALANCER] } configs { # role configs are placed in the instance groups HUE { HUE_LOAD_BALANCER { listen: 8889 } } } }   ... View more

Rather than restoring from a snapshot, could you stop and start the instance instead? You wouldn't need to reconfigure anything in Director using this method. While stopped, you could re-attach new EBS volumes created from your snapshot. See https://www.cloudera.com/documentation/director/latest/topics/director_server_startstop.html   If this workflow doesn't work for you, could you provide more information about your use case?     ... View more

TestUser,   Cloudera Director 2.6.x is configured with repositories for CM/CDH 5.13 by default. See the documentation for how to configure a non-default repository. You can look for the releant section in the instructions for any of the cloud providers. https://www.cloudera.com/documentation/director/latest/topics/director_deployment_requirements.html#concept_sm5_hxt_lt   In addition to setting the CDH product version, you need to specify the parcel repository. Please see the links to the reference conf file in my previous reply. There is a field you can fill out if you are creating a cluster through the UI. At the bottom of that page, you can open up the "Configurations" and check "Override default parcel repositories". You can specify http://archive.cloudera.com/cdh5/parcels/5.12/ and any other parcel repos you need.   David     ... View more

TestUser,   Cloudera Manager is compatible with older versions of CDH. https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#cm_cdh_compatibility   Do you have the exact error? Does Cloudera Director throw the error or does Cloudera Manager?   You specify the CDH version to Director by specifying both the product version and the parcels repository. https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf#L794 https://github.com/cloudera/director-scripts/blob/master/configs/aws.reference.conf#L804 A mismatch between these fields would cause an issue while bootstrapping.   David ... View more

Andy,   The best place to look is the agent install logs and the agent logs. /tmp/scm_prepare_node.<Unique ID> /var/log/cloudera-scm-agent   You should also check that your security group allows full access from other cluster instances (e.g., from other instances in the same security group).   It also looks like you are using custom DNS, but I still see the .ec2.internal addresses in the HostEndpoint list. If you've set up your DHCP Option Set to point to your own DNS server then you should disable DNS Hostnames and/or DNS Resolution on your VPC.   David ... View more

Tauqeer,   Cloudera Director's API will provide what you need to add instances and replace data nodes. Master migration is a little trickier. Cloudera Director can help with creating instances for the migration, but you will have to perform the migration in Cloudera Manager. See https://www.cloudera.com/documentation/director/latest/topics/director_create_ha_clusters.html#migrate_hdfs_master_roles for information on how to use Cloudera Director when migrating masters.   These tasks will mainly use Cloudera Director's "update" API. See https://github.com/cloudera/director-sdk/blob/master/java-client/src/main/java/com/cloudera/director/client/latest/api/ClustersApi.java#L549 or https://github.com/cloudera/director-sdk/blob/master/python-client/cloudera/director/latest/ClustersApi.py#L625   There are examples in both java and python to get you started. https://github.com/cloudera/director-sdk ... View more

David,   Cloudera Director 2.5 changed the way it tags instances to include the tags on the runInstances request. This feature has not been added to the China region. I'm afraid that you will have to stay on 2.4.x until Amazon updates that region to support this features.   A proxy server will work.   Which yum packages are timing out? Here are other alternatives.   You could create an ami with the packages pre-installed. https://github.com/cloudera/director-scripts/tree/master/faster-bootstrap   You could host the parcels and packages. If you do this, then you will need to configure the custom repos when you create a Deployment or Cluster. https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_package_repo.html https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_parcel_repo.html     Good Luck! David ... View more

Shailesh,   Thanks for bringing this up. We are aware of the problem and are actively working on a fix for the tck.   In the meantime, you can test your plugin by replacing the plugin that was installed by Cloudera Director with your own version.   David   ... View more