Hi all,
After "sucessfully" install kafka on a host with HDP 2.5.3, started the service from ambari, we are not able to produce or consum topics.
We can create and describe a topic:
-bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper zoo.host.com:2181 --create --topic newmikl --partition 1 --replication-factor 1
Created topic "newmikl".
-bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper zoo.host.com:2181 --describe --topic newmikl Topic:newmikl PartitionCount:1 ReplicationFactor:1 Configs:
Topic: newmikl Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Unfortunately in the logs.dir we can't see the tpoic files:
-bash-4.2$ ls -l /hadoop/kafka/kafka-logs/
total 4
-rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 cleaner-offset-checkpoint
-rwxrwxrwx 1 kafkauser kafkagroup 57 Mar 17 09:45 meta.properties
-rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 recovery-point-offset-checkpoint
-rwxrwxrwx 1 kafkauser kafkagroup 0 Mar 17 09:45 replication-offset-checkpoint
I case we want to produce ad consum messages, we're facing an error:
-bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list kafka.host.com:6667 --topic newmikl --security-protocol SASL_PLAINTEXT
test
test1
test2
test3
^C[2017-03-17 11:14:24,025] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
-bash-4.2$ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper zoo.host.com:2181 --topic newmikl --security-protocol SASL_PLAINTEXT --from-beginning
{metadata.broker.list=kafka.host.com:6667, request.timeout.ms=30000, client.id=console-consumer-56271, security.protocol=SASL_PLAINTEXT}
[2017-03-17 11:14:35,594] WARN Fetching topic metadata with correlation id 0 for topics [Set(newmikl)] from broker [BrokerEndPoint(1001,kafka.host.com,6667)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-03-17 11:14:35,597] WARN [console-consumer-56271_slbifregkhn01.fr.intranet-1489745675236-1a4e34e3-leader-finder-thread], Failed to find leader for Set([newmikl,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(newmikl)] from broker [ArrayBuffer(BrokerEndPoint(1001,kafka.host.com,6667))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
{metadata.broker.list=kafka.host.com:6667, request.timeout.ms=30000, client.id=console-consumer-56271, security.protocol=SASL_PLAINTEXT}
[...]
etc...
Looking kafka logs, we can see errors since the start of kafka:
# tail -f /var/log/kafka/controller.log
java.io.IOException: Connection to kafka.host.com:6667 (id: 1003 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-03-18 15:15:29,340] WARN [Controller-1003-to-broker-1003-send-thread], Controller 1003's connection to broker kafka.host.com:6667 (id: 1003 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to kafka.host.com:6667 (id: 1003 rack: null) failed
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:233)
at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Anyone have an idea about what can be wrong in our configuration?
Thanks.
