Reply
Explorer
Posts: 19
Registered: ‎12-01-2014

Shutdown and power off to move a CDH cluster physically

My rack of servers has to move to another location.  I have about 20 servers in the CDH cluster and a cloudera manager.  What is the proper process to shutdown a cluster, power off the hardware, and follow it up by bringing up the hardware followed by the OS and SW.  

 

Thanks,

Anant

Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: Shutdown and power off to move a CDH cluster physically

I use this :-)

 

for i in {1..20}; do ssh n$i shutdown -h now; done

 

Never have issues and do it all day for SUT testing.  No data loss or issues.

Cloudera Employee
Posts: 229
Registered: ‎09-23-2013

Re: Shutdown and power off to move a CDH cluster physically

Stop CDH Services through CM

Stop Management Services through CM

Stop CM

(Shutdown should handle the postgres embedded DB if used)

Shutdown system. 

Explorer
Posts: 19
Registered: ‎12-01-2014

Re: Shutdown and power off to move a CDH cluster physically

Thank you, this is more acceptable than the prior suggestion.  Yes, the postgres db is on the same machine as the Cloudera manager. The cloudera manager node is part of the rack, has to be shutdown and moved as well.  Now, I am assuming thhe bring up is the opposite.  Power on all servers, bring up CM service and then bring up the entire cluster of all services.

Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: Shutdown and power off to move a CDH cluster physically

If it's a server in production the preferred way and recommended way is the following command on each server.

 

# for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x stop ; done

 

Then make sure all services are shutdown

 

# ps -aef | grep java

 

Then issue the command on my previous message in the proper way. 

 

Explorer
Posts: 19
Registered: ‎12-01-2014

Re: Shutdown and power off to move a CDH cluster physically

Yes, they are production servers.  Now, is there a difference between the CM shutting the service versus me doing on each server through "service stop"?  Is it just convenience?

 

Now, when the servers are coming powering up wont each service come up as they are listed in the "chkconfig"?  Does anything special need to be done in the CM after the servers are powered up?

 

Thanks again,

Anant

Highlighted
Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: Shutdown and power off to move a CDH cluster physically

I use the script because some times we work from 6 to 200+ cloudera machines so it's easier :-)

 

If you make sure all services start on boot your set.  I always verify services have all started on all machines.

 

# for i in {1..2000}; do ssh n$i service {servicename} status; done

 

You can also use 

 

# for i in {1..2000}; do ssh n$i service {servicename} status & done

 

 

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: Shutdown and power off to move a CDH cluster physically

Service scripts (/etc/init.d) for CDH services don't work in a CM installation. In fact, you shouldn't even install them, and CM's host inspector will give you a warning if it detects these.

CM needs to manage processes. If you start them via service scripts, then CM doesn't know about them, and you will frequently get things like port conflicts between the CM-managed process and the service script process. Service scripts will never stop the CM-managed process.

Service scripts for CM server, db, and agent are fine to have.

Nauseous, I'm not sure how what you're doing works for you unless you simply aren't using Cloudera Manager.
Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: Shutdown and power off to move a CDH cluster physically

It's an alternative dlo which might be faster for some. I use it because the process in cloudera can be understood at the command line and then by understanding more you can also find issues, configuration changes and understand the operations better.

 

Also Cloudera website also annouces these options can be used instead of cloudera manager interface and knowing it can be faster.  All these options are well documented on Cloudera.

 

I found that cloudera interface isn't always working the way it should on settings. What I mean they don't take effect some times and using the CLI actually is a way of verifying and speeding up the process.

Cloudera Employee
Posts: 509
Registered: ‎07-30-2013

Re: Shutdown and power off to move a CDH cluster physically

Can you link the documentation you are referencing?

It is simply impossible for normal service scripts to interact with CM-managed processes. You could write custom ones that went through the CM API to do this, but the default ones definitely won't work that way. If you are not using CM, then it's of course fine to use those service scripts.

If a config isn't taking effect, that's usually because you didn't refresh / restart the appropriate role (there will be a staleness icon letting you know), or you think a config needs to go somewhere but CM doesn't think so. It's pretty common for people to think a configuration needs to go to places that don't actually need them. I'd be curious to understand a specific scenario where this is happening.

I also maintain that it is impossible to replace the CM UI with service scripts. You need to use the CM API or UI to interact with the processes that CM manages.