Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Highlighted

CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Explorer

I have been trying to use CB via the CLI and Hortonworks Data Cloud and keep getting the error mentioned in the title. The error was also mentioned in this forum topic (https://community.hortonworks.com/questions/45536/using-cloudbreak-to-create-cluster-on-aws.html)

We are just trying to make an HDP2.4 cluster using vanilla settings (internal HCatalog, default instance types and number , only changing the HDP distro number).

Is anyone hitting this now? If not, did you recently update CloudBreak? I wonder if this issue is with our environment somehow, or recent changes to CB.

5 REPLIES 5

Re: CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Rising Star

Hi @phillip young

Could you please share the link of the AWS Cloudformation template that you have used to create the HDC control plane?

We had issues on older RC AMIs, but we will try to reproduce this issue on your version.

Br,

Tamas

Highlighted

Re: CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Explorer

Sure. I attached the CF template HDC made.

Highlighted

Re: CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Explorer

It seems that the VPC we are using might need to be set up to create EC2 instances with public IPs. This worked in VPCs that HDC created, but not in an existing VPC. We are looking at different VPC parameter differences that might affect CloudBreak. We can compare the working VPC generated by the run of HDC that created a VPC (as opposed to the runs where it used an existing VPC), but if there is a list of VPC requirements, that would save us doing some testing.

Highlighted

Re: CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Rising Star

Hi @phillip young

Here you can find the documentation of HDC: http://hortonworks.github.io/hdp-aws/launch/

I think there is no detailed list of requirements for existing VPCs, but we should create it.

When we tested existing VPC based deployments "DNS resolution" and "DNS hostnames" was enabled in our VPC.

7039-screen-shot-2016-08-29-at-124355-pm.png

Br, Tamas

Highlighted

Re: CloudBreak fails with "duplicate key" error, where key is an IP (of a worker node)

Explorer

When we (I work with Phil) made the following changes, we no longer encountered the duplicate key error:

1) Changed the subnet where we were launching the clusters so that it auto-assigned public IPs.

2) Changed our VPC's DHCP option set to use the Amazon US-East default domain name (ec2.internal) instead of our internal domain name. We were using AmazonProvidedDNS instead of a custom name server, so our EC2 instances were getting assigned .ec2.internal domain names by Amazon anyway. According to Amazon, the domain name value in the DHCP options set is used to complete unqualified DNS hostnames.

Don't have an account?
Coming from Hortonworks? Activate your account here