Running Multiple Kafka Brokers on one HDP Sandbox

[#1]

Hi,

How do I go about running multiple Kafka brokers on the HDP 2.4 (or 2.5) Sandbox?

Thanks,

[#2 · Accepted solution]

@Andrew Watson

In kafka properties file you should specify, let say 3 brokers. You find that properties file in the /conf folder. Then you restart kafka service.

in HDP 2.5 sandbox, server.properties file can be found at /usr/hdp/current/kafka-broker/conf

You have to add a few lines like these and adjust for your logs location preference, or at least create the folders needed. Those ports also need to have forwarding rules in case you want to access anything from outside of your VM, or at least to be available.

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1

broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2

broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

[#3]

I want to do the same but in a cluster environment , how can I create more brokers on different nodes?

[#4]

Hi @Andrew Watson

@Constantin Stanca is right... you can start 2 or 3 different broker process, each with its own logs dir, etc... They will run as different brokers but in the same machine...

You can create a config properties for one broker, then change broker id, port and log directory. Then copy it with different config and start each broker using the different config properties. Things to look at

broker.id

logs.dirs

port (since running in same machine, need to be different)