Support Questions
Find answers, ask questions, and share your expertise

Producing to Kafka HDP Sandbox 2.6.4

Producing to Kafka HDP Sandbox 2.6.4

New Contributor

I am unable to produce message to kafka.

Using the HDP Sandbox 2.6.4 for virtual box (on mac).

  1. Imported fresh VM from .ova.
  2. Started fresh machine.
  3. Logged into Ambari as raj_ops.
  4. Started Kafka and took it out of Maintenance Mode.
  5. Started web shell http://localhost:4200
  6. Created a topic
  7. Started a the consumer
[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --zookeeper localhost:2181 --topic simpletesttopic --partitions 1 --replication-factor 1 --create
Created topic "simpletesttopic".
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
simpletesttopic
[root@sandbox-hdp kafka-broker]# bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --from-beginning --topic simpletesttopic

Then I opened a new web shell and used the console produce to attempt to send a message.

[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-console-producer.sh --topic simpletesttopic --broker-list localhost:6667
A Test Message
[2018-07-10 17:31:47,435] ERROR Error when sending message to topic simpletesttopic with key: null, value: 14 bytes with error: (org.apache.kaf
ka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

So, why doesn't this work? I didn't change anything in the sandbox besides starting Kafka. Someone will come on and tell me to check the logs, OK I'll do that... there is nothing after the topic creation. Here is the last 11 lines of the controller.log and last 4 lines of the server.log.

[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker/logs
[root@sandbox-hdp logs]# tail -11 controller.log
[2018-07-10 17:29:25,086] DEBUG [TopicChangeListener on Controller 1001]: Topic change listener fired for path /brokers/topics with children ATLAS_HOOK,simpletesttopic,ATLAS_ENTITIES,__consumer_offsets (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [TopicChangeListener on Controller 1001]: New topics: [Set(simpletesttopic)], deleted topics: [Set()], new partition replica assignment [Map([simpletesttopic,0] -> List(1001))] (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [Controller 1001]: New topic creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Controller 1001]: New partition creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Partition state machine on Controller 1001]: Invoking state change to NewPartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,096] INFO [Replica state machine on controller 1001]: Invoking state change to NewReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,098] INFO [Partition state machine on Controller 1001]: Invoking state change to OnlinePartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,100] DEBUG [Partition state machine on Controller 1001]: Live assigned replicas for partition [simpletesttopic,0] are: [List(1001)] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,102] DEBUG [Partition state machine on Controller 1001]: Initializing leader and isr for partition [simpletesttopic,0] to(Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:2) (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,116] INFO [Replica state machine on controller 1001]: Invoking state change to OnlineReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,118] INFO [Controller-1001-to-broker-1001-send-thread], Controller 1001 connected to sandbox-hdp.hortonworks.com:6667 (id:1001 rack: null) for sending state change requests (kafka.controller.RequestSendThread)

[root@sandbox-hdp logs]# tail -4 server.log
[2018-07-10 17:29:25,121] INFO [ReplicaFetcherManager on broker 1001] Removed fetcher for partitions simpletesttopic-0 (kafka.server.ReplicaFetcherManager)
[2018-07-10 17:29:25,123] INFO Completed load of log simpletesttopic-0 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2018-07-10 17:29:25,124] INFO Created log for partition [simpletesttopic,0] in /kafka-logs with properties {compression.type -> producer, message.format.version -> 0.10.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> true, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2018-07-10 17:29:25,125] INFO Partition [simpletesttopic,0] on broker 1001: No checkpointed highwatermark is found for partition [simpletesttopic,0] (kafka.cluster.Partition)
5 REPLIES 5

Re: Producing to Kafka HDP Sandbox 2.6.4

New Contributor

I am also facing same issue

Re: Producing to Kafka HDP Sandbox 2.6.4

Rising Star

@Eric RichardsonHi,

Could you please share the port set in Listeners property in Kafka's server.properties file. Also I would recommend to use FQDN of broker instead of localhost in producer/consumer command.

Thank you!

Re: Producing to Kafka HDP Sandbox 2.6.4

Hey @Eric Richardson!
Try to run it with the following command:

bin/kafka-console-producer.sh --topic simpletesttopic --broker-list 0.0.0.0:6667

If does it work, change the LISTENER property inside the server.properties under the KAFKA_HOME (or through Ambari). Doing what @dbains said it's a good practice! :)

If it doesn't, let's figured out through the following commands:

#Let's find who's the leader
bin/kafka-topics.sh --list --zookeeper localhost:2181 --describe --topic simpletesttopic #Check on ZK namespace as well bin/zkCli or /bin/zookeeper-client ls /brokers/topics/simpletesttopic/partitions/0
If nothing the things listed above works, let's try to add the --sync to your producer.
bin/kafka-console-producer.sh --topic simpletesttopic --broker-list 0.0.0.0:6667 --sync

PS: I didn't test the last solution, but I looked at the code.

https://github.com/apache/kafka/blob/96bcfdfc7c9aac075635b2034e65e412a725672e/core/src/main/scala/ka...

https://github.com/apache/kafka/blob/5a2fcdd6d480e9f003cc49a59d5952ba4c515a71/clients/src/main/java/...

Hope this helps!

Re: Producing to Kafka HDP Sandbox 2.6.4

Explorer
@Eric Richardson

Try to use your ip address or hostname instead of localhost, you can check if the port 6667 is associated to your IP or 0.0.0.0 using the command below.

Netstat -atnp | grep 6667

Re: Producing to Kafka HDP Sandbox 2.6.4

New Contributor

It's worked for me after changing my listener ip from localhost to exact ip of virtualbox.