Altus Director 6.2: Failure to connect to Azure instance with public IP during deployment

Question

We have a need to enable public IPs on the master nodes due to an requirement in Azure when adding instances to a load balancer pool. Our cluster is accessed via private IPs only, outside of this requirement.

Within the cluster bootstrap file, we enable public IPs for the master instance groups. During deployment, director will throw an error similar to the following for some but not all master nodes:

', errorInfo=ErrorInfo{code=INSTANCE_SSH_PORT_UNAVAILABLE, properties={sshServiceEndpoints=[BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='10.0.14.15', hostAddress=Optional.of(/10.0.14.15)}, port=Optional.absent(), url=Optional.absent()}, BaseServiceEndpoint{hostEndpoint=HostEndpoint{hostAddressString='52.232.245.168', hostAddress=Optional.of(/52.232.245.168)}, port=Optional.absent(), url=Optional.absent()}]}, causes=[]}}

[2019-07-10 02:49:45.507 +0000] INFO [p-d4ed238466ce-WaitForSshToSucceed] 5f79b1c9-b7d6-41d0-a933-0cfff86cf6e4 POST /api/d6.2/environments/azuresb/deployments/azuresb-1/clusters com.cloudera.launchpad.bootstrap.WaitForServersUntilTime - com.cloudera.launchpad.bootstrap.WaitForServersUntilTime: Waiting until 2019-07-10T03:09:45.038Z for an accessible port on endpoints [10.0.14.15:22, 40.79.57.122:22]

From the logs, it look like Director should try both IP's. I confirmed that I can ssh to the instance's private IP from Director - which leads me to believe it tried to SSH via the public IP, failed, and never tried to connect via the private IP. As I mentioned, this does not happen for all masters, however all masters are configured identically - with a private and public IP.

Can someone confirm if both IP's would need to be accessible from the Director server for the installation to continue?

dturner · Accepted Answer

We discovered that there were sporadic network issues in the tunnel between Azure and AWS ( our director instance is in AWS).  Our assumption is that this was causing transient connection issues between director and Azure instances.  Declaring issue solved for now as we no longer are experiencing it.

dhan · Answer

You are correct in that Director will attempt to connect via each IP in the list, preferring to connect over the private IP. You should be fine if only one of the IPs is accessible. Director should cycle through the IPs until the configured timeout, default 20 minutes after instance allocation.

Is the cluster failing to bootstrap? There are some natural places where we would expect connectivity to fail in a transient manner. E.g., when the VM is first allocated but doesn't yet respond to ssh, when rebooting the VM. If you are seeing failures that are preventing the cluster from bootstrapping (or even causing individual instances to fail) then we would be interested in seeing the log files to investigate.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.