how to change IP's on all machines in ambari cluster?

[#1]

hi all,

we have ambari cluster on customer site ( version 2.6.1 )

cluster include the following machines: master01 , master02 , master03 , worker01-worker64 , kafka01 - kafka03

all machines are now with IP's 162.98.23.5-75

while customer want to change to these ip's

176.12.10.5-75

please advice what is the procedure to change IP's on master's machines , workers machines and kafka's machines ?

[#2 · Accepted solution]

@Michael Bronson

Ambari Server depends on the FQDN of the hosts (Setting IP Addresses are not good idea). Most of the configurations requires that we pass either 0.0.0.0 address (listen to all available network interfaces) OR listen to FQDN (hostname -f).

So if your configurations should be having mostly the FQDNs (like "zookeeper.connect" , "listeners" ...etc) should be using the FQDN then changing the IP Address will not have any effect. Only you will need to make the required DNS mapping changes or the "/etc/hosts" file changes.

So you can stop the services that that gets affected due to IP Address change and then make the IP Address change (either at DNS level OR in the "/etc/hosts" file) and then restart the services. (Assuming that you have not hardcoded any IP Address in the kafka or other component configurations).

Also we will need to restart the ambari agents on those hosts.

https://community.hortonworks.com/content/supportkb/48998/correct-procedure-to-re-ip-a-client-host-i...

[#3]

@Jay , something isn't clearly , lets say we want to change the IP on master01 node ( JournalNode )

the article say to "Shutdown all services" dose this mean to stop all components on master01 as App Timeline server , Hive , namenode , resource manager , Webcat ... etc ?

regarding to stop the agent its clearly but not clearly if we need to stop all componets on the machine ?

SupportKB

To change the IP address of a client node in a cluster the following procedure can be followed:

• Shutdown all services on the node including the Ambari Agent
• Update the DNS entry for the node with the new IP address
• Restart networking on the node so that it gets its new IP address. Ensure the hostname is reported correctly using
  • On the node:
    • hostname # should return the short name of the host
    • hostname -s # should return the short name of the host
    • hostname -f # should return the FQDN
  • On the Ambari Server:
    • nslookup nodename.fqdn # should return the new IP address of the node
• Start up the Ambari Agent on the node
• Check in Ambari that the node is registered and showing as up
• Start all the services on the node

[#4]

@Michael Bronson

• Shutdown all services on the node including the Ambari Agent

Maintenance Windows is required for such critical operations.

>>> This is needed because if some components are already running on that host and they have already opened some ports to a specific IPAddress (interface) then changing the IP Address might cause those services to not function properly.

It means if componentA on that host is listening on 162.98.23.5:50070 Or 162.98.23.5:8020 Or 162.98.23.50075 ...etc then i do not think that there is a safe way (without having any data corruption) to tell the components to dynamically switch the IP without a restart.

Restarting the components that are running on that host and has opened some ports should be the recommended approach.
One quick way to check this may be to find out which components are still using old IPs.

Example:

# cd /var/log/hadoop/hdfs
# grep '162.98.23.5' *

I am not aware of any other safe way which can be used to dynamically allow changing the running components IPAddress without restart.

[#5]

@Jay so if we need to stop all components on each host , then its better to stop all services on the cluster because we need to change the IP's on all hosts - do you agree ?

second do you have API that stop all services in the cluster automatically ? ( I mean all services will stoped accorording to the service priority )

@Jay for you information we want to develop automation scripts that will first stop all services and ambari-agent and then will change the IP's and then will start all services in the cluster

[#6]

@Michael Bronson

If the host where the IP Addresses are supposed to be changed are running Master components like NameNode, Resource Managers ..Hive Server/Metastore ...etc then it is better to restart the whole service.

Regarding Stop and Start all Services present in the cluster i copied the commands from: (not tested recently) But it should work well. https://community.hortonworks.com/questions/88211/rest-api-for-stopping-and-starting-all-services-on...

In order to stop all service using ambari API (On the whole cluster) you can do the following:

curl -i -u admin:admin -H "X-Requested-By: ambari"  -X PUT  -d '{"RequestInfo":{"context":"_PARSE_.STOP.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"Sandbox"}},"Body":{"ServiceInfo":{"state":"INSTALLED"}}}'  http://localhost:8080/api/v1/clusters/Sandbox/services

. In order to start all services you can do the following:

curl -i -u admin:admin -H "X-Requested-By: ambari"  -X PUT  -d '{"RequestInfo":{"context":"_PARSE_.STOP.ALL_SERVICES","operation_level":{"level":"CLUSTER","cluster_name":"Sandbox"}},"Body":{"ServiceInfo":{"state":"STARTED"}}}' http://localhost:8080/api/v1/clusters/Sandbox/services

.

[#7]

@Jay just to summary the steps

my steps are

1. stop all services in the cluster ( by API in the script)

2. stop ambari agent on all hosts

3. change the IP's on all machines - masters, workers , kafka's , ( by the script )

4. start agent on all hosts

5. start services in the cluster ( by API in the script )

do you agree with this steps?

second and very important question - dose the API that stop/start the services in the cluster do it with the right order ?

because non order service shutdown service could be problematic ( we are working on production systems )

@Jay I will happy to know your opinion about my automations procesdure ....

[#8]

@Michael Bronson

Yes, the steps looks fine to me.

However it would be good to add the following steps: (Two Additional Steps)

3A. After IP Address change (Validation) . Perform a validation steps mentioned in the article:

• On the node:
  • hostname # should return the short name of the host
  • hostname -s # should return the short name of the host
  • hostname -f # should return the FQDN
• On the Ambari Server:
  • nslookup nodename.fqdn # should return the new IP address of the node

4A. After Agent restart (Validation), If possible then Check if the Hosts are getting registered properly to the Ambari Server properly or not? Either by logging in to the Ambari UI OR using the API call. To see if they are using the correct updated IP or not?

# curl -u admin:admin -H "X-Requested-By: ambari"  -X GET http://hdfcluster1.example.com:8080/api/v1/hosts?fields=Hosts/host_name,Hosts/i

.

[#9]

@Michael Bronson Yes API call performs the service start and stop in correct order. Ambari UI basically does invoke the same API calls. When we say "Start All Services" Or ":Stop All Services" from Ambari UI.

[#10]

@Jay , first thank you so much you are really one of the best here your knowledge is impressive ,

I just thinking about case , for example what we can do in case we activate the stop API , and from some reason one or more services not stooped , do we can give the API retry option that will try to stop again in case some services not stopped ?