Re: set nodes hostname during provisioning

aarman · ‎06-27-2019

Hi GaryS, (1) Can you verify connectivity between Director host and Cloudera Manager host (maybe try pinging Cloudera Manager from Director host). The Director server logs may also contain more information on why the connection validation failed. You can refer here to see how to access the server logs: https://www.cloudera.com/documentation/director/latest/topics/director_troubleshoot.html (2) Unfortunatly, nodes can only be added/removed through the UI, it can't be done via config files. Though in your case since Director server can't reach Cloudera Manager, it would fail regardless, since this connectivity is required to properly add nodes to a cluster.

aarman · ‎04-19-2019

This probably belongs in Altus Cloud Services QA rather than the Altus Director board. https://community.cloudera.com/t5/Cloudera-Altus-Cloud-Services-Q/qa-p/Altus Yes it's possible to create a Kerberized cluster by using a 'secured' environment. You can see refer to the documentation here for more details: https://www.cloudera.com/documentation/altus/topics/alt_environment_intro.html#secure_clusters

aarman · ‎04-19-2019

Thanks dturner, I'll request to move this to the Cloudera Manager forums. First run, and the scripts associated with it are handled by Cloudera Manager, so they may have more context on whether such hacks are needed.

aarman · ‎04-18-2019

Hi dturner, I just attempted this and wasn't able to reproduce the issue. My cluster creation which included Hue succeeded using CM+CDH 6.2 with CentOS7.6. In fact I couldn't even find the offending package (wrapt) anywhere on the Hue instance, so there's something different in our environments/installation. Maybe more information about the enviornment/configuration would help narrow down the problem (whether your using preloaded/prextracted images, AWS or Azure, any special bootstrap scripts, etc)

aarman · ‎10-08-2018

@Baris Glad you got it working. With Director version 2.4+ you actually don't need to use a bootstrap script as shown in my previous example. You can just specify the CSD URL on the conf file and Director will automatically download and place it in ' /opt/cloudera/csd/'. See the documentation here: https://www.cloudera.com/documentation/director/latest/topics/director_non-cdh_products_custom_descriptors.html#concept_e2y_dpj_4z

aarman · ‎08-03-2018

Glad you got it working. Just to note something like this did work for me for hvm instance types: hvm=c5.large,\ c5.xlarge,\ c5.2xlarge,\ c5d.2xlarge There should be a ec2.virtualizationmappings.properties.example file you can look at, modify as an example.

aarman · ‎08-02-2018

Hmm, so etc/ec2.virtualizationmappings.properties is the default location to specify the virtualization mapping. You can use etc/ec2.customvirtualizationmappings.properties as shown in your example but then you would have to specify this custom location in etc/aws-plugin.conf (as mentioned in the docs). Can you double check to see if that's the issue.

aarman · ‎07-31-2018

These new instance types aren't in Director yet. For now however, you should be able to update Director's AWS plugin to include them. You can refer to the documentation for the steps: https://www.cloudera.com/documentation/director/latest/topics/director_aws_new_instance_types.html

aarman · ‎07-05-2018

As mentioned by dhan, you can ssh into a cluster instance and run the following. sudo yum --version If that command succeds (returns exit code 0), Director should be able to identify the package manager.

aarman · ‎07-05-2018

You can find Cloudera Management Service properties here: https://www.cloudera.com/documentation/enterprise/properties/5-15-x/topics/cm_props_mgmtservice.html

aarman · ‎03-22-2018

A keypair corresponding to the private key you supplied to Director needs to exist in AWS. Can you verify that the key pair exists for that region by checking the EC2 console. If not, you can refer to the docs here to import the public key.

aarman · ‎03-22-2018

Director doesn't have a built in way to specify hostnames like that. You may be able to achieve something like this through the use of bootstrapScripts which lets you run a script on the instance. Something to keep in mind, Director requires forward and reverse hostname resolutions in order to work correctly, so you will probably have to mess with DNS settings as well. This thread here may be of help.

aarman · ‎03-22-2018

Thanks for reporting this jmccormick. Your right, that 5.13 repo has an incorrect manifest, we're looking into fixing this. In the meantime you can try http://archive.cloudera.com/cdh5/parcels/5.13.2 which worked fine for me.

aarman · ‎12-08-2017

That conf looks right to me. Though the service should be SPARK2_ON_YARN instead of SPARK2-ON_YARN. Also can you verify that you hava Java 8 installed which is a requirement for Spark 2, see here for additional details.

aarman · ‎12-08-2017

Any helpful information in cloudera-director-server.out ?

aarman · ‎12-07-2017

It's possible to define the security group ids and subnet id at the instance template level (same place where instance type is defined). I'm not sure if there is a sample you can refer to since most Director setups place all cluster instances in the same subnet and security group. Something to keep in mind when setting up the network is that Director requires SSH access to all instances and we generally expect full connectivity among cluster and CM instances. If you want to restrict specific ports, you can refer to the Cloudera Manager docs to see which ports are needed by CM and CDH.

aarman · ‎12-07-2017

Is this reproducible? Director does cache some of the parcel repository information but it gets cleared at the beginning of bootstrap so that shouldn't be the problem. You can also try restarting Director (which also clears the cache) just to make sure.

aarman · ‎12-07-2017

Which OS are you using? Also can you check the Director server log if it's present, it should be at /var/log/cloudera-director-server/application.log

aarman · ‎12-07-2017

Also since your dealing with multiple masters, I assume your attempting to create a High Availability cluster which is a more complex setup. You can refer to this doc and this sample reference for more information on HA.

aarman · ‎12-07-2017

Are you trying to add more than 1 HDFS Balancer role? Based on the exception message it looks like that role needs to be in a group that has 1 node.

aarman · ‎12-05-2017

1. Yes you can place the EBS properties in that section of the conf file. See here on the available EBS properties. 2. The KAFKA_BROKER role should not be placed in a gateway group, there should be a Kafka GATEWAY role for this. The sample reference configuration file provided when Director is installed (can also find here) has a fairly comprehensive list of services and roles available. Although it looks like this doc needs to be updated since it's missing the gateway role for Kafka. 3. Yes, having these gateway roles should propagate the necessary configs and allow using the cli. 4. No, Director doesn't allow placing cluster services in the CM instance. 5. Yes, Director attempts to sync with Cloudera Manager every few minutes, so you should be abel to make direct changes in CM as long as the change is consistant accross the instance group. Refer here for additional details.

aarman · ‎12-05-2017

From reading the Cloudera Manager docs, it looks like these directories will be automatically created.

aarman · ‎12-05-2017

Unfortunatly at the moment there is no way to update the scripts for an existing instance. An alternate option, although a bit painful, is to create a new instance group (with the same roles and configs) with the new bootstrap scripts and then delete the old instance group. However this will not work for the post create scripts since these are not specified at the instance level.

aarman · ‎11-24-2017

1. No, Cloudera Director can't point to existing instances, the EC2 instances must be provisioned by Director. 2. Based on my answer to (1), I don't think this question still applies. In general though, since Director manages provisioning and preparing the instances as well as installing Cloudera Manager and CDH, you'll likely run into much less compatibility issues using Director than if you were doing any of these steps manually. 3. Director can install Streamset via CSD. You can refer to this post to see an example using a bootstrap script. Note that as of Director 2.5 it's actually easier now to specify CSDs and doesn't require a bootstrap script, this is documented here.

aarman · ‎10-26-2017

What version of Cloudera Manager are you on? Director should be picking up both service and role configurations as long as you are using Cloudera Manager 5.12 or higher. Also note that Director syncs up with CM every 5 minutes so you should ensure that you've waited that long to verify that the config changes were detected.

aarman · ‎10-26-2017

Unfortuantly, this is a known limitation of Hocon, see https://github.com/lightbend/config/issues/122. Glancing at the comments, it doesn't look like there are any workarounds to get this type of substitution working.

aarman · ‎10-26-2017

include file(...) should work but I believe it requires the full file path. The current latest version of Director (Director 2.6) is on typesafe config version 1.2 which doens't have support for required. You can refer to the following spec to see what's supported in this verison: https://github.com/lightbend/config/blob/v1.2.0/HOCON.md

aarman · ‎09-15-2017

You may want to check whether CM is up and running properly. It's possible that CM is out of memory which can happen when the CM instance size is too small.

aarman · ‎07-31-2017

Looking at the known issues, this isn't direclty supported in Director.

aarman · ‎06-19-2017

You may need to use SOCKS Proxy See here for additional details: https://www.cloudera.com/documentation/director/latest/topics/director_get_started_azure_socks.html

Online	Offline
Last Visited	‎07-03-2019 03:23 PM

Date Registered	‎08-24-2015 12:26 PM
Last Visited	‎07-03-2019 03:23 PM
Total Messages Posted	38
Total Kudos Received	10

Re: set nodes hostname during provisioning

Re: CDH 5.13 manifest / directory mismatch

Re: Bootstrap of cluster fails due to attempt to u...

Re: Setup cloudera director, manager and clusters ...

Re: Architecting an CDH cluster on AWS for some PB...

Re: Adding nodes to a cluster post creation via an...

Re: Cloudera Altus Cloud - Kerberized Cluster

Re: Permission errors loading Python modules: CDH6...

Re: Permission errors loading Python modules: CDH6...

Re: Sreamsets setup on CDH 5.9.1

Re: cloudera director c5d instance type support

Re: cloudera director c5d instance type support

Re: cloudera director c5d instance type support

Re: Cluster bootstrap failed: Unsupported Package ...

Re: Cloudera Director config overrides in template

Re: Cloudera Director setup

Re: set nodes hostname during provisioning

Re: CDH 5.13 manifest / directory mismatch

Re: How do I activate a parcel in a Cloudera Direc...

Re: Failed to start Cloudera Director Server. Retu...

Re: security group for the cluster and edges on AW...

Re: Bootstrap of cluster fails due to attempt to u...

Re: Failed to start Cloudera Director Server. Retu...

Re: cloudera director GUI fail to bootstrap the cl...

Re: cloudera director GUI fail to bootstrap the cl...

Re: roles for different service on cloudera

Re: installing kudu using cloudera director

Re: How to update bootstrap script post deployment

Re: Setup cloudera director, manager and clusters ...

Re: Error when cloning HA (configured) HDFS cluste...

Re: Referencing environment variables with include...

Re: Using include statement in Cloudera Director H...

Re: MonitorDaemon-Reporter throttling_logger ERROR...

Re: Cloudera Director supports on KTS/KMS installa...

Re: Cloudera Enterprise Data Hub on Azure