I have created a github repo https://github.com/mehuljn/kafka_sandbox4 with a vagrant file which can help with the details of the configuration. If I use the host machine (not the guest) and connect to the "forwarding ports, 19092/29092/39092" as configured in Vagrantfile to List or create topics it works ; but I am unable to consumer or produce. Has anyone tried to make kafka consumer and producer work in such an environment ? I am trying to make it work by using Haproxy , not sure if thats possible or not. Any inputs or direction would be helpful. Updated Error Details As I mentioned when I create or list topics it works(not from guest vm but from host ). I didn't mention earlier but I do see Connection refused warning as below Mehuls-MacBook-Pro:kafka_2.11-0.9.0.0 mehuljani$ bin/kafka-topics.sh --create --zookeeper localhost:12181 --replication-factor 1 --partitions 1 --topic test2 Created topic "test2".
Mehuls-MacBook-Pro:kafka_2.11-0.9.0.0 mehuljani$ bin/kafka-topics.sh --list --zookeeper localhost:32181
[2016-02-29 11:57:22,254] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
And while producing messages from console producer it gives the errors below on the host. Mehuls-MacBook-Pro:kafka_2.11-0.9.0.0 mehuljani$ bin/kafka-console-producer.sh --broker-list localhost:19092 --topic test
[2016-02-29 12:03:58,057] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: Batch Expired (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
[2016-02-29 12:03:58,058] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: Batch Expired (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
[2016-02-29 12:04:03,295] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: Batch Expired (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
... View more
I have case where we need to move data from one topic to other topic. I saw a utility in Kafka documentation "ReplayLogProducer". Its supposed to be run as indicated below. bin/kafka-run- class .sh kafka.tools.ReplayLogProducer Does this tool require the partitions on source topic same as that of destination partitions? How does the retention of data work on the new topic? It would be great if anyone can provide any insight on any best practices to be followed or caveats to keep in mind while running this tool.
... View more