In migrating HDP to the cloud I wanted to gain insight into what other have done or if there is a best practice relating to communications. I have seen some article which talk about using internal AWS IP addresses etc but wanted answer for the following scenario.
Is there a need to have dual IPs for a Master/Worker host for communications i.e should there be a subnet for inter cluster traffic and one for other traffic? This is with reference to a platform that is completely in the private space so no external over the internet traffic.
Or AWS having elastic throughput to an extent or using placement groups to reserve throughput would a more simple architecture just placing the master/worker nodes in one subnet with one ipaddress allocated to each node be sufficient?
Hi @Niraj Saparia,
You need to first understand how site to site tunnel configured in your network for accessing the AWS instances .The IP which you are using to connect the AWS instance from your LAN that only you can use in host file while configuring the cluster.
Let me know if I clear your doubt.
The user connection would be over AWS direct connect so effectively private. My question is should the master/worker nodes communicate in two different sinners ie 1 for cluster comms and the other for other traffic? Or is there no benefit ?