question Re: ,Solr/hbase configuration from Cloudera director in Archives of Support Questions (Read Only)

,Solr/hbase configuration from Cloudera director

kartikbha — Fri, 16 Sep 2022 10:20:03 GMT

Using Cloudera aws.reference.conf I want to enable below services :

1. enable Solr, Indexers, lilly indexer.

2. set HA Zookeepers service

3. enable standby master for Hbase.

Re: ,Solr/hbase configuration from Cloudera director

Jayita — Tue, 17 May 2016 15:58:37 GMT

Hi Kartik,

In the configuration file that you use for bootstra,

* The list of services should include SOLR, KS_INDEXER, ZOOKEEPER, HBASE

* You would need to define 3 master roles

* The SERVER role for zookeeper needs to be on all 3 (for ZK HA/quorum)

* The MASTER role for hbase needs to be on 2 master roles

* For KS Indexer, add the role HBASE_INDEXER to any 1 master (e.g. KS_INDEXER: [HBASE_INDEXER])

* For solr, add the role SOLR_SERVER to any 1 master (e.g. SOLR: [SOLR_SERVER])

Note that the masters role groups need to have unique names, e.g. masters-1 {}, masters-2 {} and so on.

Re: ,Solr/hbase configuration from Cloudera director

kartikbha — Tue, 17 May 2016 17:04:49 GMT

Please review below details.

##KS_INDEXER: [HBASE_INDEXER]&SOLR: [SOLR_SERVER]toonlymaster1
##HBASE: [MASTER]to2masters
##ZOOKEEPERallmasters
#################master1
masters-1{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]SOLR: [SOLR_SERVER]KS_INDEXER: [HBASE_INDEXER]
}
}
#################master2
masters-2{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}
}
################master3
masters-3{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}

Re: ,Solr/hbase configuration from Cloudera director

kartikbha — Wed, 18 May 2016 17:21:54 GMT

Can you please review it.

Re: ,Solr/hbase configuration from Cloudera director

Jayita — Wed, 18 May 2016 17:33:01 GMT

Yes, sure. I'll take a look today and get back to you.

Re: ,Solr/hbase configuration from Cloudera director

kartikbha — Thu, 19 May 2016 05:03:28 GMT

Hi Jayita,

Can you please also add below point.

HDFS availability can be accomplished by deploying the NameNode with high availability with at least three JournalNodes.

Regards,

Kartik Bhatnagar

Re: ,Solr/hbase configuration from Cloudera director

Jayita — Thu, 19 May 2016 18:47:27 GMT

Hi Kartik,

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

Hope this helps...

Re: ,Solr/hbase configuration from Cloudera director

kartikbha — Sun, 22 May 2016 08:19:16 GMT

Thanks Jayita for sharing HA Git hub link.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

Below is some part copied form above conf file.

masters-1 {
count: 2

instance: ${instances.i22x} {
tags {
group: masters-1
}
}

roles {
ZOOKEEPER: [SERVER]
HIVE: [HIVESERVER2, HIVEMETASTORE]
YARN: [RESOURCEMANAGER]
HBASE: [MASTER]
HUE: [HUE_SERVER]
OOZIE: [OOZIE_SERVER]
}

is it correct to have two HS2 and HMS running services at two masters since count is 2 so it will be started at two places.

Regards,

Kartik

Re: ,Solr/hbase configuration from Cloudera director

Jayita — Mon, 23 May 2016 05:14:37 GMT

Hi, Are you hitting any errors when you set up with the hive roles created twice? 2 metastores are needed for HA and having two HS2 is also a valid configuration.