I am unable to produce message to kafka.
Using the HDP Sandbox 2.6.4 for virtual box (on mac).
- Imported fresh VM from .ova.
- Started fresh machine.
- Logged into Ambari as raj_ops.
- Started Kafka and took it out of Maintenance Mode.
- Started web shell http://localhost:4200
- Created a topic
- Started a the consumer
[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --zookeeper localhost:2181 --topic simpletesttopic --partitions 1 --replication-factor 1 --create
Created topic "simpletesttopic".
[root@sandbox-hdp kafka-broker]# bin/kafka-topics.sh --list --zookeeper localhost:2181
ATLAS_ENTITIES
ATLAS_HOOK
__consumer_offsets
simpletesttopic
[root@sandbox-hdp kafka-broker]# bin/kafka-console-consumer.sh --bootstrap-server localhost:6667 --from-beginning --topic simpletesttopic
Then I opened a new web shell and used the console produce to attempt to send a message.
[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker
[root@sandbox-hdp kafka-broker]# bin/kafka-console-producer.sh --topic simpletesttopic --broker-list localhost:6667
A Test Message
[2018-07-10 17:31:47,435] ERROR Error when sending message to topic simpletesttopic with key: null, value: 14 bytes with error: (org.apache.kaf
ka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
So, why doesn't this work? I didn't change anything in the sandbox besides starting Kafka. Someone will come on and tell me to check the logs, OK I'll do that... there is nothing after the topic creation. Here is the last 11 lines of the controller.log and last 4 lines of the server.log.
[root@sandbox-hdp ~]# cd /usr/hdp/current/kafka-broker/logs
[root@sandbox-hdp logs]# tail -11 controller.log
[2018-07-10 17:29:25,086] DEBUG [TopicChangeListener on Controller 1001]: Topic change listener fired for path /brokers/topics with children ATLAS_HOOK,simpletesttopic,ATLAS_ENTITIES,__consumer_offsets (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [TopicChangeListener on Controller 1001]: New topics: [Set(simpletesttopic)], deleted topics: [Set()], new partition replica assignment [Map([simpletesttopic,0] -> List(1001))] (kafka.controller.PartitionStateMachine$TopicChangeListener)
[2018-07-10 17:29:25,089] INFO [Controller 1001]: New topic creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Controller 1001]: New partition creation callback for [simpletesttopic,0] (kafka.controller.KafkaController)
[2018-07-10 17:29:25,092] INFO [Partition state machine on Controller 1001]: Invoking state change to NewPartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,096] INFO [Replica state machine on controller 1001]: Invoking state change to NewReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,098] INFO [Partition state machine on Controller 1001]: Invoking state change to OnlinePartition for partitions [simpletesttopic,0] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,100] DEBUG [Partition state machine on Controller 1001]: Live assigned replicas for partition [simpletesttopic,0] are: [List(1001)] (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,102] DEBUG [Partition state machine on Controller 1001]: Initializing leader and isr for partition [simpletesttopic,0] to(Leader:1001,ISR:1001,LeaderEpoch:0,ControllerEpoch:2) (kafka.controller.PartitionStateMachine)
[2018-07-10 17:29:25,116] INFO [Replica state machine on controller 1001]: Invoking state change to OnlineReplica for replicas [Topic=simpletesttopic,Partition=0,Replica=1001] (kafka.controller.ReplicaStateMachine)
[2018-07-10 17:29:25,118] INFO [Controller-1001-to-broker-1001-send-thread], Controller 1001 connected to sandbox-hdp.hortonworks.com:6667 (id:1001 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
[root@sandbox-hdp logs]# tail -4 server.log
[2018-07-10 17:29:25,121] INFO [ReplicaFetcherManager on broker 1001] Removed fetcher for partitions simpletesttopic-0 (kafka.server.ReplicaFetcherManager)
[2018-07-10 17:29:25,123] INFO Completed load of log simpletesttopic-0 with 1 log segments and log end offset 0 in 0 ms (kafka.log.Log)
[2018-07-10 17:29:25,124] INFO Created log for partition [simpletesttopic,0] in /kafka-logs with properties {compression.type -> producer, message.format.version -> 0.10.1-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000000, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> true, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
[2018-07-10 17:29:25,125] INFO Partition [simpletesttopic,0] on broker 1001: No checkpointed highwatermark is found for partition [simpletesttopic,0] (kafka.cluster.Partition)
