I'm trying to add a remote host, to my local CDH installation. There is a router (gateway) between the CDH 6.3 cluster and the remote Cloud provider. The CDH cluster is sitting on a local IP VLAN that's separate from the local IP VLAN used by the Cloud hosts.
| CDH Cluster: Local IP VLAN: 10.0.x.x | -> (Gateway: PUB IP: 100.X.X.X ) -> | AWX, GCP, AWS, OVH, etc. : Local IP VLAN: 10.1.X.X |
I also have three-layer SSL / TLS working locally on the CDH cluster.
I managed to get through a plethora of Certificate issues to get it to what I think might be close to getting a successful addition to CM. But I'm stuck on the part where it tries to do a reverse lookup on the connecting SSH_CLIENT.
Where's the issue?
The issue is when the remote CM is using the reverse lookup on the connecting IP.
[root@cm-awn01 ~]# w -i 08:46:10 up 36 min, 1 user, load average: 0.03, 0.02, 0.05 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT cdhroot pts/0 220.127.116.11 08:10 2.00s 0.03s 0.00s sshd: cdhroot [priv] [root@cm-awn01 ~]#
This IP is the external IP of the gateway router, so naturally, the reverse lookup of this IP will be different then what I have in my /etc/cloudera-scm-agent/config.ini.
cat /etc/cloudera-scm-agent/config.ini | grep host server_host=srv-c01.cdh.local.hst
The script scm_prepare_node.sh then proceeds to change the server_host to that of the reverse lookup of IP 18.104.22.168:
cat /etc/cloudera-scm-agent/config.ini | grep host server_host=dhcp-100-0-0-100.remote.user.isp.com
This is a problem because the SSL certificates only contain three hosts in the SAN field:
WrongHost: Peer certificate subjectAltName does not match host, expected dhcp-100-0-0-100.remote.user.isp.com, got DNS:srv-c01.cdh.local.hst, DNS:cm-r01nn01.cdh.local.hst, DNS:cm-r01nn02.cdh.local.hst
If the script didn't reconfigure the config.ini, there wouldn't be a problem. I have the correct ports open and mapping complete both through the GW Firewall and local on the remote server via /etc/hosts .
This is a bit of a unique use case here. The purpose is to POC the addition of remote workers and resources to a locally configured Cloudera Cluster.
What to do?
So I want to know what could I do in this situation. One way to solve this would be to regenerate the SSL certificates to include the remote hostname as part of the SAN field. (This has obvious longer term issues.) The other, I suppose is to use some sort of VPN and then add the machine.
But I'm trying to figure out if there isn't a way to control what IP script scm_prepare_node.sh receives. Currently it uses SSH_CLIENT to determine the hostname. Or perhaps there is a way to prevent the script from reconfiguring the config.ini?
Is there a way to control the host addition behaviour above?