Reply
Highlighted
Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Reducing set of two master group of each 3 to one.

Required advice on few important points. 

 

  • As cloudera recommends to use two different sets of masters (one for hdfs and another for remaining one) Can’t we manage all masters using 3-4 machines instead of 6 machines? (ref#1)
  • Second how to do HA for Solr. ref#1 is set of masters, where to start extra solr server for HA.
  • As we are using cloudera director client. Can we create AMI of launcher once cluster is created and save it for future upgrade. That way we will be able to save cost of idle launcher and restart using created AMI whenever upgrade required to cluster. 

 

(ref#1)

services: [HDFS, YARN, ZOOKEEPER, HBASE, HIVE, HUE, OOZIE, SENTRY, SPARK_ON_YARN,SOLR,KS_INDEXER,HBASERESTSERVER,HBASETHRIFTSERVER]


hdfsmasters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: hdfsmasters-1
}
}
roles {
HDFS: [NAMENODE, FAILOVERCONTROLLER, JOURNALNODE]
}

# NameNode nameservice, autofailover, and quorum journal name must be configured for high availability
configs {
HDFS {
NAMENODE {
dfs_federation_namenode_nameservice: hanameservice
autofailover_enabled: true
dfs_namenode_quorum_journal_name: hanameservice
}
}
}
}

hdfsmasters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: hdfsmasters-2
}
}
roles {
HDFS: [JOURNALNODE, HTTPFS] # HTTPFS role needed for HUE
}
}

masters-1 {
count: 2
instance: ${instances.i2x} {
tags {
group: masters-1
}
}
roles {
ZOOKEEPER: [SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE]
YARN: [RESOURCEMANAGER]
HBASE: [MASTER,HBASETHRIFTSERVER,HBASERESTSERVER]
HUE: [HUE_SERVER]
OOZIE: [OOZIE_SERVER]
}
# Oozie plugins must be configured for high availability
configs {
OOZIE {
OOZIE_SERVER {
oozie_plugins_list: "org.apache.oozie.service.ZKLocksService,org.apache.oozie.service.ZKXLogStreamingService,org.apache.oozie.service.ZKJobsConcurrencyService,org.apache.oozie.service.ZKUUIDService"
}
}
}
}

masters-2 {
count: 1
instance: ${instances.i2x} {
tags {
group: master
}
}
roles {
ZOOKEEPER: [SERVER]
YARN: [JOBHISTORY]
HBASE: [HBASETHRIFTSERVER] # HBASETHRIFTSERVER role needed for HUE
SOLR: [SOLR_SERVER]
KS_INDEXER: [HBASE_INDEXER]
}
}

 

 

Regards,

Kartik