Director on AWS with custom DNS fails

[#1]

I am using cloudera director 2.6 in AWS to build a CM/CDH 5.13 cluster.  When using the default Amazon DNS the cluster builds and executes properly - although I have no resolution back to my on-prem services. 

Our desire is to have the AWS instances on our corporate DNS so we built a linux AMI that is setup to resolve to our on-prem DNS, and I am using this as the AMI for our cluster instances.  I have changed the settings in AWS to have DHCP options set to point to the same on-prem DNS - additionally I set the 'DNS Resolution' setting to NO.  The ec2 instances created have a name ip_X_X_X_X@ourcompany.com as expected, however there is no resolution for these names.

As a result of theses setting - when i use cloudera director to create a cluster it fails as it is trying to install the agent on the CM Server instance (failure to resolve the hostname).  It appears that we need an automated way to add these new ec2 instances into our on-prem DNS, or is there another solution we are overlooking? 

Has anyone else been successful in using a similar configuration?  Do you use a tool to aumoate the entry of instances into the on-prem DNS?   

Thanks for any insights you can provide!

[#2]

Sentara,

You are correct that your DNS must be able to resolve the ip-X-X-X-X@ourcompany.com names or whatever names you want your hosts to use. Definitely review AWS's documentation https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html

Here's some suggestions:

1. You can run a bootstrap script to your InstanceTemplates to add your hostname to DNS using dhclient or NetworkManager if your DNS server is configured to allow updates from your VPC instances. We have published some scripts to help people using Cloudera Director in Azure. You can refer to and adapt these scripts to your network. See https://github.com/cloudera/director-scripts/tree/master/azure-dns-scripts

2. You can brute force the problem by pre-registering all potential hostnames in your VPC or subnet with your DNS server. The subnets are configured to use IP addresses in a specified range and the hostnames are derived from those IP addresses in a known, systematic way. Thus, you can produce the expected hostnames for each IP address in your subnet and add those to your zone files.

3. You can investigate using a bootstrap script to obtain a company-specific hostname that is resolvable in your company DNS and configure your instance to use that hostname instead of theip-X-X-X-X hostname.

Please report back any progress or solutions that work for you to share the knowledge with the rest of the community.

If anyone else in the community has any suggestions, please chime in!

Thanks,

David

[#3]

Thank you David for the reply.  We were able to get a working solution using nsupdate.  Within the first boot section of the instance AMI we are setting the instance hostname to a custome name that includes the last 2 sections of the IP address (note that this needs to be the full domain name including company.com for cluster to build properly), and then doing a nsupdate to enter the hostname into the DNS.  Both A and PTR records need to be entered for each instance.  Within the AWS VPC we have the DHCP Options set to our company DNS server and DNS Resolution and DNS Hostname both set to NO.  

Cheers!

[#4]

Sentara,

Thank you very much for posting your approach here. I'm sure that it will help future community users.

David