Support Questions

Find answers, ask questions, and share your expertise

Guidelines on Replacing Nodes in Hadoop cluster running on Azure VM's

avatar
Rising Star

We are having a 6 node cluster ( 2 master and 3 slave nodes and 1 Edge Node) on AWS VM's deployed using Ambari. I need to replace the old VM's with VM's of increased size and memory.. What would be the best startegy t o replace the existing 3 slave nodes with a new set of slave nodes with a different instance type. Since we have some data in HDFS. what would be the best strategy to retain the data and bring up the cluster with new nodes.

Will it be advisable to replace Namenode or just Slave Nodes and Edge Node?

1 ACCEPTED SOLUTION

avatar
Master Mentor

you should add new nodes to the cluster and rebalance cluster, then decommission each node one at a time. Since you're replacing three nodes and that's all of your hdfs, it make take some time. Consider larger footprint as you know hdfs maintains replication factor of 3. You're best suited to use a tool like

Cloudbreak or if you're only running ETL, discovery, data science workloads, you can try Hortonworks Data Cloud. Both can add and remove instances as well as provision new instances with new machine type easily.

View solution in original post

14 REPLIES 14

avatar
Rising Star

@Artem Ervits

Yes thats what my understanding is. Thanks for confirming.

By any chance is there a plan in future for cloudbreak to deploy cluster on already running VM's in cloud?

avatar
Master Mentor
@rahul gulati

not sure, I'm not privy to the Cloudbreak roadmap. Perhaps you'd want to open this as a new HCC question and someone from the Cloudbreak team can respond?

avatar
Rising Star

@Artem Ervits

Ok sure. Just wanted to know about Namenode as well in case that also needs an upgrade to VM of higher memory and CPU. What should be the ideal scenario in that case?

Thanks

avatar
Master Mentor

Add two new nodes, then in Ambari there's an option to move namenode in the HDFS section.

avatar
New Contributor

@Artem Ervits

> Both can add and remove instances as well as provision new instances with new machine type easily.

Could you please point where that option could be located in the UI or CLI of Cloudbreak? Thank you!