question CDH 5.1 host IP address change in Archives of Support Questions (Read Only)

CDH 5.1 host IP address change

AmitD — Tue, 31 Oct 2017 15:58:21 GMT

I have a CDH 5.1 cluster with 3 nodes. We installed it using cloudera manager automated installation. It was running perfect until we moved the box to a different network and IP addresses changed. I tried following steps

1. Stopped service, cloudera-scm-server.
2. Stopped service, cloudera-scm-agent
3. Edit the /etc/cloudera-scm-agent/config.ini
4. change the server host to the new ip.
5. restart service, cloudera-scm-agent, cloudera-scm-server.

not working .

Then i followed http://www.cloudera.com/content/cloudera/en/documentation/cloudera-manager/v4-latest/Cloudera-Manager-Administration-Guide/cmag_change_hostnames.html Not helped even after changing the ips in the PostgreSQL directly.

I found following blog : http://www.geovanie.me/changing-ip-of-node-in-cdh-cluster/

Getting following error in the scm-agent log file

ProtocolError: <ProtocolError for 127.0.0.1/RPC2: 401 Unauthorized>

No helped .... Can anyone please help how to change all IP addresses in a cdh 5.1 cluster safely .....

Thanks, Amit

Re: CDH 5.1 host IP address change

AmitD — Wed, 15 Oct 2014 20:10:08 GMT

I found the solution for the above issue

1) for protocolError we need to use kill -9 to kill the superviser

2) the above mention steps are correct I was making some typo with the ip and hostnames in the config.ini in one of the hosts.

Hope that help....

Thanks,

Amit

Re: CDH 5.1 host IP address change

sanjaysubramani — Sun, 24 Jan 2016 22:05:31 GMT

I recently returned my Comcast router and put my own. The IP addresses changed and my cluster is down 😞

I am running 5.4.7CDH (starving developers version) on Ubuntu 12.04 LTS

1NN + 3DN

I followed these instructions and managed to get the IPs of the nodes corrected ojn the cluster (by updating HOSTS table in local cloudera db)

http://shulhi.com/change-ip-address-for-existing-nodes-in-cdh-5-3/

I made sure that the UUIDs of each node as displayed in Cloudera manager is same as

/var/lib/cloudera-scm-agent/uuid

Now My cloudera agents don't start on the nodes

/var/log/cloudera-scm-agent/cloudera-scm-agent.out

[24/Jan/2016 13:37:38 +0000] 14519 MainThread agent INFO SCM Agent Version: 5.4.7

[24/Jan/2016 13:37:38 +0000] 14519 MainThread agent INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log

Traceback (most recent call last):

File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3233, in <module>

main()

File "/usr/lib/cmf/agent/src/cmf/agent.py", line 3216, in main

agent.configure()

File "/usr/lib/cmf/agent/src/cmf/agent.py", line 428, in configure

raise Exception("No server_host line was found in agent's configuration. One of --standalone or --master must be specified.")

Exception: No server_host line was found in agent's configuration. One of --standalone or --master must be specified.

Re: CDH 5.1 host IP address change

jjjjjjhao — Sun, 22 Oct 2017 02:40:00 GMT

Thank you for this excellent solution.

In my situation, I created 1 master and 4 slaves cloudera cluster using private IP with EC2 and then store then as AMI (images). I then created newinstances from these 5 AMI images, with all installations of cloudera on them.

Then I folllowed all your steps to change the IP to current private IPs.

However, when I finally check the state of the cloudera:

sudo service cloudera-scm-agent status sudo service cloudera-scm-master status both return active (exited) Then I checked the /var/log/cloudera-scm-agent/cloudera-scm-agent.log

And there are errors like this ERROR Failed to connect to previous supervisor.

ERROR Failed rack peer update: [Errno 111] Connection refused

ERROR 14794 Monitor-HostMonitor throttling_logger

ERROR Failed to collect NTP metrics

ERROR Unexpected exception during download

Do you know why this happens? ( I followed all yoursteps)

Re: CDH 5.1 host IP address change

jjjjjjhao — Sun, 22 Oct 2017 02:41:25 GMT

Maybe I need to reinstall the cloudera manager?

Re: CDH 5.1 host IP address change

lionelsubedi — Tue, 24 Oct 2017 15:52:48 GMT

Steps to find and kill the process:

1) Find the port which is used by supervisor:
> ps aux |grep supervisor

2) kill the port

>sudo kill -9 <port number>

Re: CDH 5.1 host IP address change

bgooley — Tue, 24 Oct 2017 17:02:49 GMT

@jjjjjjhao,

The bits of errors provided don't tell enough of the story to indicate what may be wrong.

I would run: service cloudera-scm-agent restart and then see what happens in the agent log.

Also, what is the actual problem? What is wrong in Cloudera Manager, etc. It is unclear what you are trying to do or see and what actually happens. Once that is clarified, the community can help.

Ben