Reply
Expert Contributor
Posts: 66
Registered: ‎12-24-2015
Accepted Solution

,Solr/hbase configuration from Cloudera director

[ Edited ]

 

Using Cloudera aws.reference.conf I want to enable below services :

 

1. enable Solr, Indexers, lilly indexer.

 

2. set HA Zookeepers service

 

3. enable standby master for Hbase.

 

 

Cloudera Employee
Posts: 26
Registered: ‎09-21-2015

Re: ,Solr/hbase configuration from Cloudera director

Hi Kartik,

 

In the configuration file that you use for bootstra,

* The list of services should include SOLR, KS_INDEXER, ZOOKEEPER, HBASE

* You would need to define 3 master roles

* The SERVER role for zookeeper needs to be on all 3 (for ZK HA/quorum)

* The MASTER role for hbase needs to be on 2 master roles

* For KS Indexer, add the role HBASE_INDEXER to any 1 master (e.g. KS_INDEXER: [HBASE_INDEXER])

* For solr, add the role SOLR_SERVER to any 1 master (e.g. SOLR: [SOLR_SERVER])

 

Note that the masters role groups need to have unique names, e.g. masters-1 {}, masters-2 {} and so on.

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: ,Solr/hbase configuration from Cloudera director

Please review below details.

 


##KS_INDEXER: [HBASE_INDEXER]&SOLR: [SOLR_SERVER]toonlymaster1
##HBASE: [MASTER]to2masters
##ZOOKEEPERallmasters
#################master1
masters-1{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]SOLR: [SOLR_SERVER]KS_INDEXER: [HBASE_INDEXER]
}
}
#################master2
masters-2{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HBASE: [MASTER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}
}
################master3
masters-3{
count: 1instance: ${
instances.i22x
}{
tags{
group: master
}
}roles{
HDFS: [NAMENODE,
SECONDARYNAMENODE]YARN: [RESOURCEMANAGER,
JOBHISTORY]ZOOKEEPER: [SERVER]HIVE: [HIVESERVER2,
HIVEMETASTORE]HUE: [HUE_SERVER]OOZIE: [OOZIE_SERVER]SENTRY: [SENTRY_SERVER]SPARK_ON_YARN: [SPARK_YARN_HISTORY_SERVER]KAFKA: [KAFKA_BROKER]
}

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: ,Solr/hbase configuration from Cloudera director

Can you please review it.
Cloudera Employee
Posts: 26
Registered: ‎09-21-2015

Re: ,Solr/hbase configuration from Cloudera director

Yes, sure. I'll take a look today and get back to you.

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: ,Solr/hbase configuration from Cloudera director

 

Hi Jayita,

 

Can you please also add below point.

 

  • HDFS availability can be accomplished by deploying the NameNode with high availability with at least three JournalNodes.

 

Regards,

Kartik Bhatnagar

Highlighted
Cloudera Employee
Posts: 26
Registered: ‎09-21-2015

Re: ,Solr/hbase configuration from Cloudera director

Hi Kartik,

 

Regarding the sample configs you sent,

- You don't need to repeat all the master roles on each master. Most of them only need to be on the first master, just the extra ones for HA need to be on additional nodes

- If using kafka, please make sure the appropriate product version, and a repository url corresponding to the same version, are specified in the config file

- where the master role assignments differ, the group names also need to be unique (for eg the group name for masters-1 and masters-2)

 

For setting up HDFS HA, we have a sample config file to help with the role assignments. This would cover examples for some of your other HA questions as well.

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Hope this helps...

 

Expert Contributor
Posts: 66
Registered: ‎12-24-2015

Re: ,Solr/hbase configuration from Cloudera director

[ Edited ]

Thanks Jayita for sharing HA Git hub link.

 

https://github.com/cloudera/director-scripts/blob/master/configs/aws.ha.reference.conf

 

Below is some part copied form above conf file.

 
masters-1 {
  count: 2
   
  instance: ${instances.i22x} {
  tags {
  group: masters-1
  }
  }
   
  roles {
  ZOOKEEPER: [SERVER]
  HIVE: [HIVESERVER2, HIVEMETASTORE]
  YARN: [RESOURCEMANAGER]
  HBASE: [MASTER]
  HUE: [HUE_SERVER]
  OOZIE: [OOZIE_SERVER]
  }
  

 

is it correct to have two HS2 and HMS running services at two masters since count is 2 so it will be started at two places.

 

 

Regards,

Kartik

Cloudera Employee
Posts: 26
Registered: ‎09-21-2015

Re: ,Solr/hbase configuration from Cloudera director

Hi, Are you hitting any errors when you set up with the hive roles created twice? 2 metastores are needed for HA and having two HS2 is also a valid configuration.

Announcements