Support Questions
Find answers, ask questions, and share your expertise

Running Multiple Kafka Brokers on one HDP Sandbox

Solved Go to solution

Running Multiple Kafka Brokers on one HDP Sandbox

Hi,

How do I go about running multiple Kafka brokers on the HDP 2.4 (or 2.5) Sandbox?

Thanks,

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Running Multiple Kafka Brokers on one HDP Sandbox

@Andrew Watson

In kafka properties file you should specify, let say 3 brokers. You find that properties file in the /conf folder. Then you restart kafka service.

in HDP 2.5 sandbox, server.properties file can be found at /usr/hdp/current/kafka-broker/conf

You have to add a few lines like these and adjust for your logs location preference, or at least create the folders needed. Those ports also need to have forwarding rules in case you want to access anything from outside of your VM, or at least to be available.

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1
broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2
broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

View solution in original post

3 REPLIES 3
Highlighted

Re: Running Multiple Kafka Brokers on one HDP Sandbox

@Andrew Watson

In kafka properties file you should specify, let say 3 brokers. You find that properties file in the /conf folder. Then you restart kafka service.

in HDP 2.5 sandbox, server.properties file can be found at /usr/hdp/current/kafka-broker/conf

You have to add a few lines like these and adjust for your logs location preference, or at least create the folders needed. Those ports also need to have forwarding rules in case you want to access anything from outside of your VM, or at least to be available.

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1
broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2
broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

View solution in original post

Highlighted

Re: Running Multiple Kafka Brokers on one HDP Sandbox

Master Collaborator

I want to do the same but in a cluster environment , how can I create more brokers on different nodes?

Highlighted

Re: Running Multiple Kafka Brokers on one HDP Sandbox

Rising Star

Hi @Andrew Watson

@Constantin Stanca is right... you can start 2 or 3 different broker process, each with its own logs dir, etc... They will run as different brokers but in the same machine...

You can create a config properties for one broker, then change broker id, port and log directory. Then copy it with different config and start each broker using the different config properties. Things to look at

broker.id

logs.dirs

port (since running in same machine, need to be different)