question Re: Move Role to another host in Support Questions

Move Role to another host

adam990 — Mon, 17 Apr 2017 04:30:55 GMT

I have 2 host in my hadoop cluster. But one host using default Cloudera Manger Installer get more Roles. So, Memory usage from that host is high (3.6 GiB / 3.9 GiB). Can I move some roles from that host to another host that already in the cluster? If i can, What roles that i can move to another host?

Thanks

Re: Move Role to another host

Fawze — Mon, 17 Apr 2017 05:54:08 GMT

Hi Adam,

Basically you can move any role you need to the other node.

Can you please list the roles in the default node?

Re: Move Role to another host

SanjeevkishoreY — Mon, 17 Apr 2017 15:23:08 GMT

Hi Fawze,

i have 6 servers in my environment out of which 2 are commissioned few days back.

now out of 6servers in one server we have 25 roles assigned to it..now i want to migrate these roles so that the cluster should be in a balanced state..i am listing you the roles that are assigned on each server..please help me in balancing the cluster and also please give me the commands to run the balancer on the cluster.

here are the roles that are currently runing on the cluster.

HDFS DataNode	HDFS DataNode	HBase RegionServer	HBase Thrift Server		HBase RegionServer		HBase RegionServer
		HDFS DataNode	HBase Master	HDFS DataNode		HDFS DataNode
Disk Usage	Disk Usage	Hive Gateway	HBase RegionServer	Hive Gateway		Hive Gateway
2.5 TB/22.7 TB	2.5 TiB / 22.7 TiB	Impala Daemon	HDFS Balancer	Impala Daemon		Impala Daemon
		Kafka Broker	HDFS NameNode	Kafka Broker		Kafka Broker
Physical Memory	Physical Memory	Solr Server	HDFS SecondaryNameNode	Solr Server		Solr Server
10.6 GB/ 252 GB	11.9 GiB / 252.2 GiB	Spark Gateway	Hive Gateway	Spark Gateway		Spark Gateway
		YARN (MR2 Included) NodeManager	Hive Metastore Server	YARN (MR2 Included) NodeManager		YARN (MR2 Included) NodeManager
Swap space	Swap space		HiveServer2	ZooKeeper Server		ZooKeeper Server
0 GB /32 GB	0 B / 32 GiB	Disk Usage	Hue Server
		1.6 TiB / 3.2 TiB	Impala Catalog Server	Disk Usage		Disk Usage
CDH Version : CDH 5	CDH Version : CDH 5		Impala Daemon	2.3 TiB / 4.4 TiB		2.2 TiB / 4.4 TiB
OS : Redhat6.6	OS : Redhat6.6	Physical Memory	Impala StateStore
		119.1 GiB / 252.2 GiB	Cloudera Management Service Alert Publisher	Physical Memory		Physical Memory
			Cloudera Management Service Event Server	198.3 GiB / 1.5 TiB		162.4 GiB / 504.6 GiB
		Swap space	Cloudera Management Service Host Monitor
		0 B / 32 GiB	Cloudera Management Service Service Monitor	Swap space		Swap space
			Oozie Server	0 B / 32 GiB		0 B / 32 GiB
		CDH Version : CDH 5	Solr Server
		OS : Redhat6.6	Spark Gateway	CDH Version : CDH 5		CDH Version : CDH 5
			Spark History Server	OS : Redhat6.6		OS : Redhat6.6
			YARN (MR2 Included) JobHistory Server
			YARN (MR2 Included) NodeManager
			YARN (MR2 Included) ResourceManager
			ZooKeeper Server

			Disk Usage
			3.1 TiB / 4.4 TiB

			Physical Memory
			254.5 GiB / 1.5 TiB

			Swap space
			0 B / 32 GiB

			CDH Version : CDH 5
			OS : Redhat6.6

Re: Move Role to another host

mbigelow — Mon, 17 Apr 2017 16:13:07 GMT

As @Fawze mentioned, any role can be moved. From the info provide it looks like you are running 5 worker nodes and 1 master. Depending on the hardware you can make one or two of those workers into Masters; otherwise moving master roles onto worker nodes will potentially cause issues.

Some tips for moving, if the role connects to a backend DB, like the Hive Metastore, then you need to update the DB user account to reflect it running from the new hosts. Some other roles are tricky, like the Namenode, as you would need to migrate the metadata.

The two candidates to convert to masters from workers would be the two that just have the Datanode role on them. You can just do one if you don't need production level resiliency and redundancy in the master services.

First, remove the Nodemanager, HBase Regionserver, and Impala Daemon, Solr server role from the current master (the one with 25 roles).
Those are all worker roles. I would move the ResourceMananger, Spark History, Impala Catalog, Impala Statestore, and Secondary Namenode. The last one is the only tricky one and it should be fine as setting up a new one should sync from the Namenode.

I also recommend moving the Cloudera Manager roles off to a completely different server, possible a VM. It doesn't need to be on the cluster, is using up valuable resources, and can live in a VM. You will need to update all DB accounts related to them.

That is a start, we could go further but that should be enough to give you some breathing room.

Good luck.

Re: Move Role to another host

Fawze — Mon, 17 Apr 2017 17:19:48 GMT

I would recommend the following:

a- the 2 DataNodes:

1- HDFS DataNode
2- YARN (MR2 Included) NodeManager
3- Impala Daemon
4- Spark Gateway ( to overcome some bugs in different cloudera manager versions)

b- All Cloudera manager services to be on VM machine with 16-24 GB memory will be enough

c- Hive roles,Oozie, Hue, and the database ( mysql) to be on another VM as it allow you to create new one in case of disaster.

d- 2 other physical servers, both have:

1- YARN (MR2 Included) ResourceManager
2- HDFS NameNode
3- ZooKeeper Server

one of them has the following additional roles:

4- YARN (MR2 Included) JobHistory Server
5- Spark History Server
6- HDFS Balancer

and the other to have the following roles:

7-Impala Catalog Server
8- Impala StateStore

e- i would recommend small node for the HDFS,Spark and Hive Gateway, if not it can be added to the DataNodes.

Re: Move Role to another host

SanjeevkishoreY — Mon, 17 Apr 2017 18:34:11 GMT

Thanks for your reply....how to add reassign the roles to the other servers.can i do it by using the cloudera manager or i have to delete the service or add it again ..or can i stop the services by using CM and move the roles to other servers with the help of ui..

Kindly please let me know the steps how to proceed..

Thanks and Regards,

Sanjeev kishore.

Re: Move Role to another host

Fawze — Mon, 17 Apr 2017 18:54:42 GMT

Hi Sanjeev,

Each role should be migrated. moved separaetly, for example:

Name node, journal node and the failover controller:

https://www.cloudera.com/documentation/enterprise/5-6-x/topics/admin_nn_migrate_roles.html

Cloudera manager:

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_ag_restore_server.html

zookeeper:

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_mc_zookeeper_service.html#ReplacingZkServerCM

for datanode:

make sure to deco the service on the nodes you want to stop the service on, wait till the under replicated block to closed and then delete the role from the nodes

adding datanode role to another node can be done by cloudera manager by going to the HDFS ---> instances and add role instances

for NodeManager, only stop and add the same as in the HDFS

regarding hive and oozie, you need to stop the services, backup the database and then create the database on the new node.

If it not yet aproduction environment, i would recommend to create it from scratch as it safe, easier and faster

Re: Move Role to another host

saranvisa — Mon, 17 Apr 2017 19:41:29 GMT

@SanjeevkishoreY

Most of the points are covered by others, my 2 cents are

1. Configure NameNode and SecondaryNameNode on different servers. Make sure both the server has similar configuration
2. Configure Cloudera manager, Hive, Impala on different servers.
3. If possible keep Cloudera manager and YARN Resource manager in different nodes

Re: Move Role to another host

adam990 — Tue, 18 Apr 2017 13:07:05 GMT

Hi Fawze,

Sorry for my late respose and thanks for your reply.

I have 2 hosts and already move Zookeeper Server, Hue Server, Hive Gateway and Hive Metastore Server successfully via Instance pages in Cloudera Manager. All Service Started and Status Good Health except HDFS. There is an Error from HDFS: Bad : 658 under replicated blocks in the cluster. 658 total blocks in the cluster. Percentage under replicated blocks: 100.00%. Critical threshold: 40.00%

Result from : # $ hdfs fsck / -files -blocks -locations

Status: HEALTHY
Total size:   530731534 B
Total dirs:   2293
Total files:   662
Total symlinks:       0
Total blocks (validated):   658 (avg. block size 806582 B)
Minimally replicated blocks:   658 (100.0 %)
Over-replicated blocks:   0 (0.0 %)
Under-replicated blocks:   658 (100.0 %)
Mis-replicated blocks:       0 (0.0 %)
Default replication factor:   2
Average block replication:   1.0
Corrupt blocks:       0
Missing replicas:       658 (50.0 %)
Number of data-nodes:       1
Number of racks:       1
FSCK ended at Tue Apr 18 07:16:43 UTC 2017 in 426 milliseconds

The filesystem under path '/' is HEALTHY

There is Missing replicas: 658 (50.0 %)

My HDFS Replication Factor is set up to 2 and run hdfs dfs -setrep 2 /, but still not solved.

Maybe you have solutions for this.

Thanks

Re: Move Role to another host

Fawze — Tue, 18 Apr 2017 13:30:18 GMT

Hi @adam990,

As you can see from the report that you have 1 DataNode Live and not 2.

Number of data-nodes: 1

Can you please make sure that you see 2 Live DataNodes at the HDFS

Re: Move Role to another host

adam990 — Tue, 18 Apr 2017 15:02:08 GMT

Hi @Fawze,

Great, It's Work for me. So update my Default replication factor : 1. This is temporary because i need to upgrade my server memory.

Missing replicas looks normal and HDFS status is good Health

Status: HEALTHY
Total size:   530731534 B
Total dirs:   2846
Total files:   662
Total symlinks:       0
Total blocks (validated):   658 (avg. block size 806582 B)
Minimally replicated blocks:   658 (100.0 %)
Over-replicated blocks:   0 (0.0 %)
Under-replicated blocks:   0 (0.0 %)
Mis-replicated blocks:       0 (0.0 %)
Default replication factor:   1
Average block replication:   1.0
Corrupt blocks:       0
Missing replicas:       0 (0.0 %)
Number of data-nodes:       1
Number of racks:       1
FSCK ended at Tue Apr 18 14:57:18 UTC 2017 in 573 milliseconds

The filesystem under path '/' is HEALTHY

Thanks Fawze.

Re: Move Role to another host

airhead — Wed, 20 Jun 2018 20:51:08 GMT

Fawze

You said that any role can be moved. How is that done for ones like "HDFS Balancer" or JobHistory servers? These roles can't be added/moved via CM that I can see.