Kafka multi broker setup through ambari

[#1]

i configure multibroker kafka through ambari but failed to startup the kafka broker. however, it was successful when tried doing it through command line (with diff protocol,diff portnos, diff nodes)

and now i'm very much confused about the listeners property because by default amabri bound to 6667 port, what should be used for other listeners if im configuring multibroker.

all most i tried all the combinations but non of them helped me while starting it from ambari and it gives me unable to connect to zookeeoer.

example :

1. ( diff protocol, diff portnos, diff nodes) - this worked from command line but not through ambari

2.(diff protocol, diff portnos,same node) - this worked from command line but not through ambari

3. its not working neither command line nor ambari if this has been set to 'same protocol' or 'same port' -

please help me

sathish

[#2]

just to add, there are no issues if its single node broker through ambari see the below log details , but Issues are occurring only if its multi node broker.

[2016-12-20 05:40:04,272] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 13 milliseconds. (kafka.coordinator.GroupMetadataManager)

[2016-12-20 05:40:04,345] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,348] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,352] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2016-12-20 05:40:04,366] INFO Creating /brokers/ids/1001 (is it secure? true) (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,394] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,396] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(10.246.131.99,6667,PLAINTEXTSASL) (kafka.utils.ZkUtils)

[2016-12-20 05:40:04,402] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

thanks,

sathish

[#3]

@sathish jeganathan

which HDP version you are running with? there is no need to set advertised.listeners from ambari until you are running with mulitple interfaces on the same host.

in ambari you should have only listener setting like this

SASL_PLAINTEXT://localhost:6667

it looks you modified server.properties manually thats why you are seeing advertise.listener setting there

[#4]

version is 2.2.2 and i havent added any new properties in server.properties, all ive done is added new hosts with portno on existing properties.

thanks,

sathish

[#5]

can you please guide me in setting up multinode brokers...

what are the properties should be changed while setting up the multi-broker

thanks,

sathish

[#6]

also, when i create new kafka broker through ambari the config parameters are still be referring to the first kafka-broker... means, listeners and advertised.host.name are referring to the old/first broker hostname/Ip...

i'm getting error "unable to connect to zookeeper" if i add multi nodes on those properties,

thanks,

sathish

[#7]

now i'm able to bring up the brokers on diff nodes through command line but ambari still has issues.. i think, i will have to re-visit my configurations on ambari-kafka.. bcoz its not taking the listeners and hostname properties for multinode. i will have to see how can i setup properties for each individual kafka-brokers...

may be i ill have to set the listeners and hostname to 0.0.0.0:6667...

thanks,

sathish

[#8 · Accepted solution]

@sathish jeganathan I will suggest you don't Install or configure kafka manually, Install using Ambari it will take care all the settings for you, after installation once broker come online then only you can try changing configuration.

[#9]

yes.. i removed hard coded hostnames and changed it to localhost:6667(default one), restarted services from ambari.. its running fine now..

thank you very much for the help @Rajkumar Singh

[#10]

@Satish Bomma glad to see it working, could you please accept best answer in this thread so that other user can get benefit from it.

[#11]

@Rajkumar Singhsure.. will do that ... i have one more query (may be a silly one ) .. how can i browse the topic or can i see the content of the topic ?

is there any specific tool to do so or can it be done through HDFS itself ?

thanks,

sathish