I have been trying to use CB via the CLI and Hortonworks Data Cloud and keep getting the error mentioned in the title. The error was also mentioned in this forum topic (https://community.hortonworks.com/questions/45536/using-cloudbreak-to-create-cluster-on-aws.html)
We are just trying to make an HDP2.4 cluster using vanilla settings (internal HCatalog, default instance types and number , only changing the HDP distro number).
Is anyone hitting this now? If not, did you recently update CloudBreak? I wonder if this issue is with our environment somehow, or recent changes to CB.
Could you please share the link of the AWS Cloudformation template that you have used to create the HDC control plane?
We had issues on older RC AMIs, but we will try to reproduce this issue on your version.
It seems that the VPC we are using might need to be set up to create EC2 instances with public IPs. This worked in VPCs that HDC created, but not in an existing VPC. We are looking at different VPC parameter differences that might affect CloudBreak. We can compare the working VPC generated by the run of HDC that created a VPC (as opposed to the runs where it used an existing VPC), but if there is a list of VPC requirements, that would save us doing some testing.
Here you can find the documentation of HDC: http://hortonworks.github.io/hdp-aws/launch/
I think there is no detailed list of requirements for existing VPCs, but we should create it.
When we tested existing VPC based deployments "DNS resolution" and "DNS hostnames" was enabled in our VPC.
When we (I work with Phil) made the following changes, we no longer encountered the duplicate key error:
1) Changed the subnet where we were launching the clusters so that it auto-assigned public IPs.
2) Changed our VPC's DHCP option set to use the Amazon US-East default domain name (ec2.internal) instead of our internal domain name. We were using AmazonProvidedDNS instead of a custom name server, so our EC2 instances were getting assigned .ec2.internal domain names by Amazon anyway. According to Amazon, the domain name value in the DHCP options set is used to complete unqualified DNS hostnames.