Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: The Cloudera Community will undergo maintenance on Saturday, August 17 at 12:00am PDT. See more info here.

,Solr/hbase configuration from Cloudera director

SOLVED Go to solution

,Solr/hbase configuration from Cloudera director

Explorer

 

Using Cloudera aws.reference.conf I want to enable below services :

 

1. enable Solr, Indexers, lilly indexer.

 

2. set HA Zookeepers service

 

3. enable standby master for Hbase.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ,Solr/hbase configuration from Cloudera director

Contributor

Hi Kartik,

 

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

 

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Hope this helps...

 

8 REPLIES 8

Re: ,Solr/hbase configuration from Cloudera director

Contributor

Hi Kartik,

 

In the configuration file that you use for bootstra,

* The list of services should include SOLR, KS_INDEXER, ZOOKEEPER, HBASE

* You would need to define 3 master roles

* The SERVER role for zookeeper needs to be on all 3 (for ZK HA/quorum)

* The MASTER role for hbase needs to be on 2 master roles

* For KS Indexer, add the role HBASE_INDEXER to any 1 master (e.g. KS_INDEXER: [HBASE_INDEXER])

* For solr, add the role SOLR_SERVER to any 1 master (e.g. SOLR: [SOLR_SERVER])

 

Note that the masters role groups need to have unique names, e.g. masters-1 {}, masters-2 {} and so on.

Re: ,Solr/hbase configuration from Cloudera director

Explorer

Please review below details.

 


##KS_INDEXER: [HBASE_INDEXER]&SOLR: [SOLR_SERVER]toonlymaster1
##HBASE: [MASTER]to2masters
##ZOOKEEPERallmasters
#################master1
masters-1{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]SOLR: [SOLR_SERVER]KS_INDEXER: [HBASE_INDEXER]
}
}
#################master2
masters-2{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}
}
################master3
masters-3{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}

Re: ,Solr/hbase configuration from Cloudera director

Explorer
Can you please review it.

Re: ,Solr/hbase configuration from Cloudera director

Contributor

Yes, sure. I'll take a look today and get back to you.

Highlighted

Re: ,Solr/hbase configuration from Cloudera director

Explorer

 

Hi Jayita,

 

Can you please also add below point.

 

  • HDFS availability can be accomplished by deploying the NameNode with high availability with at least three JournalNodes.

 

Regards,

Kartik Bhatnagar

Re: ,Solr/hbase configuration from Cloudera director

Contributor

Hi Kartik,

 

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

 

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Hope this helps...

 

Re: ,Solr/hbase configuration from Cloudera director

Explorer

Thanks Jayita for sharing HA Git hub link.

 

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Below is some part copied form above conf file.

 
masters-1 {
  count: 2
   
  instance: ${instances.i22x} {
  tags {
  group: masters-1
  }
  }
   
  roles {
  ZOOKEEPER: [SERVER]
  HIVE: [HIVESERVER2, HIVEMETASTORE]
  YARN: [RESOURCEMANAGER]
  HBASE: [MASTER]
  HUE: [HUE_SERVER]
  OOZIE: [OOZIE_SERVER]
  }
  

 

is it correct to have two HS2 and HMS running services at two masters since count is 2 so it will be started at two places.

 

 

Regards,

Kartik

Re: ,Solr/hbase configuration from Cloudera director

Contributor

Hi, Are you hitting any errors when you set up with the hive roles created twice? 2 metastores are needed for HA and having two HS2 is also a valid configuration.