question Re: Kafka multi broker setup through ambari in Support Questions

Kafka multi broker setup through ambari

Sathish Jeganathan — Mon, 19 Dec 2016 13:49:43 GMT

i configure multibroker kafka through ambari but failed to startup the kafka broker. however, it was successful when tried doing it through command line (with diff protocol,diff portnos, diff nodes)

and now i'm very much confused about the listeners property because by default amabri bound to 6667 port, what should be used for other listeners if im configuring multibroker.

all most i tried all the combinations but non of them helped me while starting it from ambari and it gives me unable to connect to zookeeoer.

example :

1. ( diff protocol, diff portnos, diff nodes) - this worked from command line but not through ambari

2.(diff protocol, diff portnos,same node) - this worked from command line but not through ambari

3. its not working neither command line nor ambari if this has been set to 'same protocol' or 'same port' -

please help me

sathish

Re: Kafka multi broker setup through ambari

rajkumar_singh — Mon, 19 Dec 2016 13:52:23 GMT

could you please post the kafka broker startup logs which is failing after setting 'same protocol' and same port.

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Mon, 19 Dec 2016 14:00:28 GMT

below are my configurations

listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667

advertised.listeners = PLAINTEXTSASL://10.246.131.99:6667, PLAINTEXTSASL://10.246.131.34:6667, PLAINTEXTSASL://10.246.131.35:6667

below is the error

[2016-12-19 05:57:09,852] FATAL (kafka.Kafka$)

java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port

at scala.Predef$.require(Predef.scala:233)

at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:906)

at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914)

at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867)

at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699)

at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696)

at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)

at kafka.Kafka$.main(Kafka.scala:58)

at kafka.Kafka.main(Kafka.scala)

Re: Kafka multi broker setup through ambari

rajkumar_singh — Mon, 19 Dec 2016 14:06:19 GMT

@sathish jeganathan

could you please try following steps

1. open ambari, open kafka configuration

2. update listeners=SASL_PLAINTEXT://localhost:6667

3.remove advartised.listeners and save configuration

4. restart kafka broker and see if helps

Re: Kafka multi broker setup through ambari

rajkumar_singh — Mon, 19 Dec 2016 15:08:09 GMT

@sathish jeganathan any luck with the above mentioned settings.

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Mon, 19 Dec 2016 16:29:04 GMT

Hi Raj - ive tried removing advertised listeners but it gets generated automatically.. also still am getting same error "Each listener must have a different port"

thanks,

sathish

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Mon, 19 Dec 2016 16:35:28 GMT

also i tried with diff host name with same protocol and i got below error

Each listener must have a different protocol

java.lang.IllegalArgumentException: requirement failed: Each listener must have a different protocol

at scala.Predef$.require(Predef.scala:233)

at kafka.server.KafkaConfig.validateUniquePortAndProtocol(KafkaConfig.scala:907)

at kafka.server.KafkaConfig.getListeners(KafkaConfig.scala:914)

at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:867)

at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:699)

at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:696)

at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)

at kafka.Kafka$.main(Kafka.scala:58)

at kafka.Kafka.main(Kafka.scala)

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Mon, 19 Dec 2016 16:40:31 GMT

and now ive changed all the protocols,port nos (9091,92,93) and started kafka through ambari but it started giving me the below error

FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 25000

thanks,

sathish

Re: Kafka multi broker setup through ambari

rajkumar_singh — Mon, 19 Dec 2016 17:17:28 GMT

@sathish jeganathanthere is no need to specify advertised listener,what hdp version you are currently running with? can you try to set following setting and see if it helps.make both listeners and advartised.listeners same.

listeners=SASL_PLAINTEXT://localhost:6667

advartised.listeners=SASL_PLAINTEXT://localhost:6667

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 13:17:35 GMT

@Rajkumar Singh

there is no parameter advertised.listeners in ambari-kafka config but it exists in server.properties. however advertised.listeners property has got the same value as "listeners" ( both has got same values).

thanks,

sathish

Re: Kafka multi broker setup through ambari

rajkumar_singh — Tue, 20 Dec 2016 13:23:33 GMT

@sathish jeganathan

which HDP version you are running with? there is no need to set advertised.listeners from ambari until you are running with mulitple interfaces on the same host.

in ambari you should have only listener setting like this

SASL_PLAINTEXT://localhost:6667

it looks you modified server.properties manually thats why you are seeing advertise.listener setting there

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 13:53:46 GMT

just to add, there are no issues if its single node broker through ambari see the below log details , but Issues are occurring only if its multi node broker.

[2016-12-20 05:40:04,272] INFO [Group Metadata Manager on Broker 1001]: Removed 0 expired offsets in 13 milliseconds. (kafka.coordinator.GroupMetadataManager)

[2016-12-20 05:40:04,345] INFO [ThrottledRequestReaper-Produce], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,348] INFO [ThrottledRequestReaper-Fetch], Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)

[2016-12-20 05:40:04,352] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)

[2016-12-20 05:40:04,366] INFO Creating /brokers/ids/1001 (is it secure? true) (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,394] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)

[2016-12-20 05:40:04,396] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: PLAINTEXTSASL -> EndPoint(10.246.131.99,6667,PLAINTEXTSASL) (kafka.utils.ZkUtils)

[2016-12-20 05:40:04,402] INFO [Kafka Server 1001], started (kafka.server.KafkaServer)

thanks,

sathish

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 13:59:03 GMT

version is 2.2.2 and i havent added any new properties in server.properties, all ive done is added new hosts with portno on existing properties.

thanks,

sathish

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 14:00:43 GMT

can you please guide me in setting up multinode brokers...

what are the properties should be changed while setting up the multi-broker

thanks,

sathish

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 14:14:05 GMT

also, when i create new kafka broker through ambari the config parameters are still be referring to the first kafka-broker... means, listeners and advertised.host.name are referring to the old/first broker hostname/Ip...

i'm getting error "unable to connect to zookeeper" if i add multi nodes on those properties,

thanks,

sathish

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Tue, 20 Dec 2016 20:09:32 GMT

now i'm able to bring up the brokers on diff nodes through command line but ambari still has issues.. i think, i will have to re-visit my configurations on ambari-kafka.. bcoz its not taking the listeners and hostname properties for multinode. i will have to see how can i setup properties for each individual kafka-brokers...

may be i ill have to set the listeners and hostname to 0.0.0.0:6667...

thanks,

sathish

Re: Kafka multi broker setup through ambari

rajkumar_singh — Tue, 20 Dec 2016 20:12:40 GMT

@sathish jeganathan I will suggest you don't Install or configure kafka manually, Install using Ambari it will take care all the settings for you, after installation once broker come online then only you can try changing configuration.

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Wed, 21 Dec 2016 13:17:09 GMT

yes.. i removed hard coded hostnames and changed it to localhost:6667(default one), restarted services from ambari.. its running fine now..

thank you very much for the help @Rajkumar Singh

Re: Kafka multi broker setup through ambari

rajkumar_singh — Wed, 21 Dec 2016 13:19:11 GMT

@Satish Bomma glad to see it working, could you please accept best answer in this thread so that other user can get benefit from it.

Re: Kafka multi broker setup through ambari

Sathish Jeganathan — Wed, 21 Dec 2016 13:53:47 GMT

@Rajkumar Singhsure.. will do that ... i have one more query (may be a silly one ) .. how can i browse the topic or can i see the content of the topic ?

is there any specific tool to do so or can it be done through HDFS itself ?

thanks,

sathish