Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka multi broker setup through ambari

avatar

i configure multibroker kafka through ambari but failed to startup the kafka broker. however, it was successful when tried doing it through command line (with diff protocol,diff portnos, diff nodes)

and now i'm very much confused about the listeners property because by default amabri bound to 6667 port, what should be used for other listeners if im configuring multibroker.

all most i tried all the combinations but non of them helped me while starting it from ambari and it gives me unable to connect to zookeeoer.

example :

1. ( diff protocol, diff portnos, diff nodes) - this worked from command line but not through ambari

2.(diff protocol, diff portnos,same node) - this worked from command line but not through ambari

3. its not working neither command line nor ambari if this has been set to 'same protocol' or 'same port' -

please help me

sathish

1 ACCEPTED SOLUTION

avatar
Super Guru

@sathish jeganathan I will suggest you don't Install or configure kafka manually, Install using Ambari it will take care all the settings for you, after installation once broker come online then only you can try changing configuration.

View solution in original post

26 REPLIES 26

avatar

just to add, there are no issues if its single node broker through ambari see the below log details , but Issues are occurring only if its multi node broker.

[2016-12-20 05:40:04,272] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 13 milliseconds. (kafka.coordinator.GroupMetadataManager)

[2016-12-20 05:40:04,345] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,348] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,352] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2016-12-20 05:40:04,366] INFO Creating /brokers/ids/1001 (is it secure? true) (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,394] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,396] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(10.246.131.99,6667,PLAINTEXTSASL) (kafka.utils.ZkUtils)

[2016-12-20 05:40:04,402] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

thanks,

sathish

avatar
Super Guru
@sathish jeganathan

which HDP version you are running with? there is no need to set advertised.listeners from ambari until you are running with mulitple interfaces on the same host.

in ambari you should have only listener setting like this

SASL_PLAINTEXT://localhost:6667

it looks you modified server.properties manually thats why you are seeing advertise.listener setting there

avatar

version is 2.2.2 and i havent added any new properties in server.properties, all ive done is added new hosts with portno on existing properties.

thanks,

sathish

avatar

can you please guide me in setting up multinode brokers...

what are the properties should be changed while setting up the multi-broker

thanks,

sathish

avatar

also, when i create new kafka broker through ambari the config parameters are still be referring to the first kafka-broker... means, listeners and advertised.host.name are referring to the old/first broker hostname/Ip...

i'm getting error "unable to connect to zookeeper" if i add multi nodes on those properties,

thanks,

sathish

avatar

now i'm able to bring up the brokers on diff nodes through command line but ambari still has issues.. i think, i will have to re-visit my configurations on ambari-kafka.. bcoz its not taking the listeners and hostname properties for multinode. i will have to see how can i setup properties for each individual kafka-brokers...

may be i ill have to set the listeners and hostname to 0.0.0.0:6667...

thanks,

sathish

avatar
Super Guru

@sathish jeganathan I will suggest you don't Install or configure kafka manually, Install using Ambari it will take care all the settings for you, after installation once broker come online then only you can try changing configuration.

avatar

yes.. i removed hard coded hostnames and changed it to localhost:6667(default one), restarted services from ambari.. its running fine now..

thank you very much for the help @Rajkumar Singh

avatar
Super Guru

@Satish Bomma glad to see it working, could you please accept best answer in this thread so that other user can get benefit from it.

avatar

@Rajkumar Singhsure.. will do that ... i have one more query (may be a silly one ) .. how can i browse the topic or can i see the content of the topic ?

is there any specific tool to do so or can it be done through HDFS itself ?

thanks,

sathish