Re: Director on AWS with custom DNS fails

Sentara · ‎02-28-2018

Thank you David for the reply. We were able to get a working solution using nsupdate. Within the first boot section of the instance AMI we are setting the instance hostname to a custome name that includes the last 2 sections of the IP address (note that this needs to be the full domain name including company.com for cluster to build properly), and then doing a nsupdate to enter the hostname into the DNS. Both A and PTR records need to be entered for each instance. Within the AWS VPC we have the DHCP Options set to our company DNS server and DNS Resolution and DNS Hostname both set to NO. Cheers!

Sentara · ‎02-20-2018

I am using cloudera director 2.6 in AWS to build a CM/CDH 5.13 cluster. When using the default Amazon DNS the cluster builds and executes properly - although I have no resolution back to my on-prem services. Our desire is to have the AWS instances on our corporate DNS so we built a linux AMI that is setup to resolve to our on-prem DNS, and I am using this as the AMI for our cluster instances. I have changed the settings in AWS to have DHCP options set to point to the same on-prem DNS - additionally I set the 'DNS Resolution' setting to NO. The ec2 instances created have a name ip_X_X_X_X@ourcompany.com as expected, however there is no resolution for these names. As a result of theses setting - when i use cloudera director to create a cluster it fails as it is trying to install the agent on the CM Server instance (failure to resolve the hostname). It appears that we need an automated way to add these new ec2 instances into our on-prem DNS, or is there another solution we are overlooking? Has anyone else been successful in using a similar configuration? Do you use a tool to aumoate the entry of instances into the on-prem DNS? Thanks for any insights you can provide!

Sentara · ‎11-09-2017

Yes, I have a spark gateway on the host and I copied hive-site.xml into /etc/spark/conf.

Sentara · ‎11-08-2017

I am having the same issue and copying the hive-site.xml did not resolve the issue for me. I am not using spark2, but the v1.6 that comes with Cloudera 5.13 - and there is no spark/hive configuration setting. Was anyone else able to figure out how to fix this? Thanks!

Sentara · ‎11-08-2017

Hello! I am seeing this same issue ... was there a resolution to your problem? Thanks.

Online	Offline
Last Visited	‎01-09-2019 08:46 AM

Date Registered	‎09-08-2017 01:31 PM
Last Visited	‎01-09-2019 08:46 AM
Total Messages Posted	6
Total Kudos Received	1

Re: Director on AWS with custom DNS fails

Director on AWS with custom DNS fails

Re: how to access the hive tables from spark-shell

Re: how to access the hive tables from spark-shell

Re: hive.metastore.schema.verification is not enab...