Created on 09-17-2018 03:53 AM - edited 08-17-2019 06:24 AM
In this article we will talk about how to setup a separate zookeeper quorum for Kafka which is managed by Ambari.
Please note at this moment Amabri only can support one ZK Quorum, in order to have two Quorum in a cluster
Steps (to setup Zookeeper Server) :- Please run all the below command in all the nodes where zookeeper will be installed manually
2. Setting up the environment variables (ease of deployment)
3. Setting up the environment with necessary Folders and permission
4. Create Zookeeper Home and Configuration Directory
5. Extract the ZooKeeper configuration files to a temporary directory and Move the files to Home Directory created previously in Step 4
6. Setup of the Configuration files
8. Initialize the ZooKeeper data directories with the 'myid' file. Create one file per ZooKeeper server, and put the number of that server in each file:
9. Finally start the zookeeper Service on all the servers
sudo -E -u zookeeper bash -c "export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR ; export ZOOCFG=zoo.cfg; source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh ; $ZOOKEEPER_HOME/bin/zkServer.sh start"
10. Validate the ZK Server status , either it will be Leader or follower
sudo -E -u zookeeper bash -c "export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR ; export ZOOCFG=zoo.cfg; source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh ; $ZOOKEEPER_HOME/bin/zkServer.sh status"
Steps (to be ran on Amabri Managed Cluster) :- This is to configure Kafka service to Point the Quorum to the newly setup ZK serves
Please update the “zookeeper.connect” to point to the new ZK servers for example “c38-node1.example-labs.com:2181,c38-node2.example-labs.com:2181,c38-node3.example-labs.com:2181” in Amabri Kafka Config section and restart the services.
Validate Kafka Services using the new ZK quorum :-
[root@c18-node2 zookeeper]# cd /usr/hdp/current/kafka-broker
[root@c18-node2 kafka-broker]# ./bin/kafka-topics.sh --create --zookeeper c38-node1.example-labs.com:2181 --replication-factor 1 --partitions 1 --topic trest_demo
Created topic "test_demo".
[root@c18-node2 kafka-broker]# ./bin/kafka-topics.sh --list --zookeeper c38-node1.example-labs.com:2181