Unable to SSH to Cluster

[#1]

Symptoms

When attempting to access a running cluster via SSH, the connection times out or is rejected

Applies To

All versions of clusters managed by Cloudera Altus

Cause

Update needed for AWS Security group, as the incoming SSH connection originating for client machine may not be included on the defined cluster's AWS Security Group incoming connections

Troubleshooting Steps

This issue can typically occur if the AWS security group configured for does not contain an updated IP from the connecting client.  

To alleviate this issue, please perform the following steps which will add the IP address(es) needed to connect to the appropriate AWS security group:

1. Login to AWS
2. Select the Appropriate Region in the top-right hand corner, (i.e. US West (Oregon)).
3. Navigate to EC2
4. Under Resources, click on Security Groups
5. Locate the corresponding Cloudera Altus Security Group configured;  For a particular cluster in question, narrow down the options filtering by Group ID.
6. Click on the Security Group, which will then list the Description of the group in the lower portion of the window.
7. Click on the Inbound tab, then Edit.
8. Click on Add Rule, select SSH as the Type which also sets TCP as the Protocol, and the Port Range to 22.  Adjust this if needed.
9. Under Source, select one of the options, My IP or Custom, depending on the usage requirements (see notes below)
10. NOTE: Please review the current incoming list if there are multiple My IP entries as some of these may be outdated and may be removed.  Please check with your AWS administrator if needed. 
11. Click on Save to save changes, which will be implemented immediately.

NOTES:

My IP is good for setting a single IP address that is identified from the existing web connection to the AWS Web UI.  This will be the external address identified from your current machine. If more flexibility is needed, please consider using Custom.

While My IP is useful for setting a single IP, there can be administration challenges when the incoming IP address is dynamically allocated, requiring an update to the AWS security group each time the client workstation's IP changes.  If the DHCP pool or list of IP addresses is know, using a Custom setting can help reduce the amount of administration required in keeping the Security Group up-to-date.

Using a Custom setting, ensure to set the CIDR block for the given range of known IP addresses.  Per AWS:

Specify a single IP address, or an IP address range in CIDR notation (for example, 203.0.113.5/32). If connecting from behind a firewall, you'll need the IP address range used by the client computers.

References

[#2]

The issue may also happen if SSH key supplied when at cluster creation time was generated using Pass Phrase. This option is not supported when trying to use SSH Proxy command and hence command can hang/timeout.