Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Running Multiple Kafka Brokers on one HDP Sandbox

avatar

Hi,

How do I go about running multiple Kafka brokers on the HDP 2.4 (or 2.5) Sandbox?

Thanks,

1 ACCEPTED SOLUTION

avatar
Super Guru

@Andrew Watson

In kafka properties file you should specify, let say 3 brokers. You find that properties file in the /conf folder. Then you restart kafka service.

in HDP 2.5 sandbox, server.properties file can be found at /usr/hdp/current/kafka-broker/conf

You have to add a few lines like these and adjust for your logs location preference, or at least create the folders needed. Those ports also need to have forwarding rules in case you want to access anything from outside of your VM, or at least to be available.

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1
broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2
broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

View solution in original post

3 REPLIES 3

avatar
Super Guru

@Andrew Watson

In kafka properties file you should specify, let say 3 brokers. You find that properties file in the /conf folder. Then you restart kafka service.

in HDP 2.5 sandbox, server.properties file can be found at /usr/hdp/current/kafka-broker/conf

You have to add a few lines like these and adjust for your logs location preference, or at least create the folders needed. Those ports also need to have forwarding rules in case you want to access anything from outside of your VM, or at least to be available.

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1
broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2
broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

avatar
Super Collaborator

I want to do the same but in a cluster environment , how can I create more brokers on different nodes?

avatar
Expert Contributor

Hi @Andrew Watson

@Constantin Stanca is right... you can start 2 or 3 different broker process, each with its own logs dir, etc... They will run as different brokers but in the same machine...

You can create a config properties for one broker, then change broker id, port and log directory. Then copy it with different config and start each broker using the different config properties. Things to look at

broker.id

logs.dirs

port (since running in same machine, need to be different)