Thanks for this info (and link) and duly noted on troubleshooting in the future! ... View more

This issue is fixed in the Altus 6.2+.   Director unable to install unlimited strength JCE with OpenJDK Director is unable to properly detect the version of OpenJDK being used and thus is unable to install the unlimited strength JCE when bootstrapping a deployment if it is requested. Cloudera Issue: DIR-8957 ... View more

@Mike Wilson On that note I managed to find another issue with the validator.   If you create a VirtualInstanceGroup similar to the python example here: https://github.com/cloudera/director-sdk/blob/master/python-client-samples/cluster.py#L159   And then change the name in the VirtualInstanceGroup to masters.1 (add an invalid character to it like a .) it will create the cluster successfully but then the user in the UI will not be able to repair nodes / grow / shrink nor clone the cluster.   It will also just gray out the continue button and provide no feedback to the user in the logs nor the UI.   Cheers! ... View more

Just checking in to see if there are any updates.   As a workaround we are using Azure MySQL backend, but we would prefer to use Azure Postgresql.  Let me know if I can provide any additional info.  Unfortunately it's a very repeatable issue.  🙂 ... View more

Hi man,   My recommendations would be to go with GCP, these days i am prepare for GCP exam.   Thanks, Rana ... View more

followup with a little more context and clarification.   I mis-spoke when I said that " the created users are not being granted the correct role to operate on the created database." In actuality, the role provided to Altus Director is not able to create a database with an owner other than itself. The database provisioning logic attempts to create a separate role and use that as the owner for each database. SUPERUSER privileges are required to create a database with a different owner . Google cloud sql does not support giving SUPERUSER privileges (see https://cloud.google.com/sql/docs/postgres/features under "Unsupported Features").   External Database Templates should work correctly if you create your own Postgres server and provide Altus Director with a SUPERUSER role. ... View more

Wow! I'm impressed. Thank you for the detailed instructions, and good work! I would like to reiterate that anyone who follows this advice does it at their own risk, just in case you missed jcpuskar's warning . 🙂 ... View more

In your first response, it looks as though the SSH credentials aren't correct, and Director is failing attempting to SSH into the instance. Are you sure you've provided the right username and key for this instance? In the second, it looks like Java has not been installed prior to CM being installed. Do you have the javaInstallationStrategy set to none for this config? Let me know. ... View more

Hi Alok,   Can you explain how you resolved the SSH key error. Am facing the same issue.   Thanks! ... View more

I am using Hive database. All the query which I am executing is select query. Cloudera version is 5.14.2. ... View more

Thank you for the reply. ... View more

I unfortunately do not see the application log -- is the link missing?   With the cluster in UPDATE_FAILED, Director is no longer able to operate on this cluster. Do you have an earlier DB backup that you would be able to restore to?   Getting out of UPDATE_FAILED is possible, but is currently only supported as a guided process through Cloudera Director Support due to how dangerous it can be. Do you have a support contract with Cloudera at the moment?   While you can add this through CM, Director will be unable to manage this host and it may make growing/shrinking difficult once the UPDATE_FAILED problem is fixed. If you add it through CM and then recover Director, Director will be able to grow and shrink the cluster, but will be otherwise unaware of that new node, and will not be able to modify it if desired.   The extra CDH parcel could be a source of grief, but I'd like to try to get by the UPDATE_FAILED problem first. ... View more

Sorry to drag up an old topic but I am hoping there is now some official guidance from Cloudera - or perhaps a reference architecture.    As soon as an organisation builds a Cloudera EDH cluster in one Amazon AWS region (or Azure or Google Cloud, or whatever) then they soon realise they might want another cluster in a different region or a different Availability Zone.    Personally I am happy sticking with one Region but multiple AZs - so I am thinking that I want one Cloudera Director, one Cloudera Manager, and separate clusters in each AZ.  ... View more

Yes, that is exactly what happened. Is there a way to adjust Director's settings so the Cloudera Manager instance will show up? ... View more

I have the beginning of Flask based REST api I was working on before my project got discontinued. I tried my best to follow the Director API routes. I'll upload it to Github and share the link here. ... View more

Sorry for the delayed reply.   Unfortunately, there is not enough info in that log snippet to determine why the agent installation failed. Do you know whether the agent installation failed on the CM instance or one of the cluster instances? If the former, look earlier in the Director log to see if there are any additional details about the specific failure. If the latter, also look in the CM log.   ... View more

Can you give me the exact AMI IDs that you are using to try out? ... View more

Yoga,   I just tried to repro your problem and I see a BYON-specific instance template popup.   One thing to note that may be confusing you is that Director requires an Type and Image field even though these concepts are not needed in BYON. Feel free to put anything you want in those fields.   If you click on the "Advanced Options", you can specify the preferred hosts for that template. Director will attempt to pick a host from that list on a best effort basis.   David   ... View more

Indeed, thank you very much ! 🙂 ... View more

I have been able to use multiple aliases with a single host in CDH. Not multiple hostnames though. What did you hosts file look. It probably is related to how Hadoop does hostname lookups. It used specific Java methods that aren't identical to the hostname CMD, at least across all platforms. ... View more

Thanks for reverting.    Any workaround for this issue ?  Like if we can edit the port in some config file or database that the director uses ?  ... View more

That is correct! You can implement your own plugin using the SPI that you've linked. That blog post is probably the most comprehensive place to start, and the docs on the SPI github are pretty solid as well. The Javadocs for the SPI can be found here: https://cloudera.github.io/director-spi/apidocs/ I'm not totally sure what you mean by which things you need to consider to build the plugin. To try to answer generally, other than the docs you've supplied, pay attention to the Javadocs for the various functions that need to be implemented. As an example: https://cloudera.github.io/director-spi/apidocs/com/cloudera/director/spi/v1/provider/ResourceProvider.html#allocate(T,%20java.util.Collection,%20int) Check out the Javadoc for the allocate function. Director expects that this be followed closely, e.g. Director expects that the plugin will clean up after itself upon failure. Does that cover things, or is there anything else you'd like clarification on? ... View more

All, I'm very sorry about this, but it looks like this thread got lost in the shuffle when we were preparing to release our most recent version of Cloudera Director. I haven't experienced this error myself, but perusing some of our internal scripts, you may be able to get around this by setting JAVA_HOME, potentially with a Director bootstrap script. Here's an untested example: echo "export JAVA_HOME=/your/java/location" >> /etc/profile.d/java_home_setup That being said, this question may be better suited for the Cloudera Manager forum. However, why set up Zookeeper using CM directly when you can set it up using Director itself? ... View more

That helped. It works now. We had no reason to expect that the instance and instances clauses were necessary for a CM that already exists and was generated by Cloudera Director. ... View more

Thank you Mike! I have a custom AMI with JDK pre-installed. ... View more

Thats awesome thanks a ton Mike. Early this morning before your mails came in - I grew impatient 🙂  as is my nature - and did give it a shot to the Cloudera director as-is-wher-is scripts.   1. Used the Cloud Formation template here     https://s3.amazonaws.com/quickstart-reference/cloudera/latest/templates/Template2-Cloudera-AWS-ExistingVPC.template   2. Created a "ClusterLauncher Instance" on AWS   3. SSH to  "C lusterLauncher Instance"     cloudera-director bootstrap cloudera/setup-default/aws.reference.conf  Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-plugins OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 Cloudera Director 2.1.0 initializing ... The configuration file aws.reference.conf is not present or cannot be read. [ec2-user@ip-10-219-178-74 ~]$ cloudera-director bootstrap cloudera/setup-default/aws.reference.conf Process logs can be found at /home/ec2-user/.cloudera-director/logs/application.log Plugins will be loaded from /var/lib/cloudera-director-plugins OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0 Cloudera Director 2.1.0 initializing ... Installing Cloudera Manager ... * Starting ..... done * Requesting an instance for Cloudera Manager .......................... done * Installing screen package (1/1) ....... done * Running custom bootstrap script on [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com] .......... done * Waiting for SSH access to [10.219.177.189, ip-10-219-177-189.us-west-2.compute.internal, 52.43.22.181, ec2-52-43-22-181.us-west-2.compute.amazonaws.com], default port 22 ..... done * Inspecting capabilities of 10.219.177.189 .......... done * Normalizing a3508870-3de7-4dc0-84a5-f69c77610c89 ..... done * Installing ntp package (1/4) ..... done * Installing curl package (2/4) ..... done * Installing nscd package (3/4) ..... done * Installing gdisk package (4/4) ..................... done * Resizing instance root partition ......... done * Mounting all instance disk drives ............ done * Waiting for new external database servers to start running ........ done * Installing repositories for Cloudera Manager ....... done * Installing oracle-j2sdk1.7 package (1/3) ..... done * Installing cloudera-manager-daemons package (2/3) ..... done * Installing cloudera-manager-server package (3/3) ...... done * Setting up embedded PostgreSQL database for Cloudera Manager ...... done * Installing cloudera-manager-server-db-2 package (1/1) ..... done * Starting embedded PostgreSQL database ...... done * Starting Cloudera Manager server ... done * Waiting for Cloudera Manager server to start ..... done * Setting Cloudera Manager License ... done * Enabling Enterprise Trial ... done * Configuring Cloudera Manager ..... done * Deploying Cloudera Manager agent ...... done * Waiting for Cloudera Manager to deploy agent on 10.219.177.189 ... done * Setting up Cloudera Management Services ............ done * Backing up Cloudera Manager Server configuration ...... done * Inspecting capabilities of 10.219.177.189 ...... done * Done ... Cloudera Manager ready. Creating cluster C5-Reference-AWS ... * Starting ..... done * Requesting 11 instance(s) in 3 group(s) ....................................... done * Preparing instances in parallel (20 at a time) .............................................................. done * Waiting for Cloudera Manager installation to complete ... done * Installing Cloudera Manager agents on all instances in parallel (20 at a time) ........ done * Waiting for new external database servers to start running ... done * Creating CDH5 cluster using the new instances ... done * Creating cluster: C5-Reference-AWS .... done * Downloading parcels: CDH-5.7.2-1.cdh5.7.2.p0.18,KAFKA-2.0.2-1.2.0.2.p0.5 ... done * Distributing parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ... done * Activating parcels: KAFKA-2.0.2-1.2.0.2.p0.5,CDH-5.7.2-1.cdh5.7.2.p0.18 ...... done * Configuring Hive to use Sentry ... done * Creating Sentry Database ... done * Calling firstRun on cluster C5-Reference-AWS ... done * Waiting for firstRun on cluster C5-Reference-AWS .... done * Running cluster post creation scripts ...... done * Adjusting health thresholds to take into account optional instances. ... done * Done ...   ... View more