Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

,Solr/hbase configuration from Cloudera director

avatar
Rising Star

 

Using Cloudera aws.reference.conf I want to enable below services :

 

1. enable Solr, Indexers, lilly indexer.

 

2. set HA Zookeepers service

 

3. enable standby master for Hbase.

 

 

1 ACCEPTED SOLUTION

avatar
Rising Star

Hi Kartik,

 

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

 

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Hope this helps...

 

View solution in original post

8 REPLIES 8

avatar
Rising Star

Hi Kartik,

 

In the configuration file that you use for bootstra,

* The list of services should include SOLR, KS_INDEXER, ZOOKEEPER, HBASE

* You would need to define 3 master roles

* The SERVER role for zookeeper needs to be on all 3 (for ZK HA/quorum)

* The MASTER role for hbase needs to be on 2 master roles

* For KS Indexer, add the role HBASE_INDEXER to any 1 master (e.g. KS_INDEXER: [HBASE_INDEXER])

* For solr, add the role SOLR_SERVER to any 1 master (e.g. SOLR: [SOLR_SERVER])

 

Note that the masters role groups need to have unique names, e.g. masters-1 {}, masters-2 {} and so on.

avatar
Rising Star

Please review below details.

 


##KS_INDEXER: [HBASE_INDEXER]&SOLR: [SOLR_SERVER]toonlymaster1
##HBASE: [MASTER]to2masters
##ZOOKEEPERallmasters
#################master1
masters-1{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]SOLR: [SOLR_SERVER]KS_INDEXER: [HBASE_INDEXER]
}
}
#################master2
masters-2{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}
}
################master3
masters-3{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}

avatar
Rising Star
Can you please review it.

avatar
Rising Star

Yes, sure. I'll take a look today and get back to you.

avatar
Rising Star

 

Hi Jayita,

 

Can you please also add below point.

 

  • HDFS availability can be accomplished by deploying the NameNode with high availability with at least three JournalNodes.

 

Regards,

Kartik Bhatnagar

avatar
Rising Star

Hi Kartik,

 

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

 

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Hope this helps...

 

avatar
Rising Star

Thanks Jayita for sharing HA Git hub link.

 

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Below is some part copied form above conf file.

 
masters-1 {
  count: 2
   
  instance: ${instances.i22x} {
  tags {
  group: masters-1
  }
  }
   
  roles {
  ZOOKEEPER: [SERVER]
  HIVE: [HIVESERVER2, HIVEMETASTORE]
  YARN: [RESOURCEMANAGER]
  HBASE: [MASTER]
  HUE: [HUE_SERVER]
  OOZIE: [OOZIE_SERVER]
  }
  

 

is it correct to have two HS2 and HMS running services at two masters since count is 2 so it will be started at two places.

 

 

Regards,

Kartik

avatar
Rising Star

Hi, Are you hitting any errors when you set up with the hive roles created twice? 2 metastores are needed for HA and having two HS2 is also a valid configuration.