Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache 2.0 Kafka from HDP 3.1 Not Producing messages

Highlighted

Apache 2.0 Kafka from HDP 3.1 Not Producing messages

Super Guru

topic creation

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bme680

nifi-app.log:2018-12-23 17:04:19,681 WARN [kafka-producer-network-thread | producer-5] org.apache.kafka.clients.NetworkClient [Producer clientId=producer-5] Error while fetching metadata with correlation id 2068 : {bme680out=LEADER_NOT_AVAILABLE}

controller.log message

[2018-12-23 17:00:06,774] INFO [Controller id=1001] Resigned (kafka.controller.KafkaController) [2018-12-23 17:00:14,112] INFO [ControllerEventThread controllerId=1001] Starting (kafka.controller.ControllerEventManager$ControllerEventThread) [2018-12-23 17:00:14,127] ERROR [ControllerEventThread controllerId=1001] Error processing event Startup (kafka.controller.ControllerEventManager$ControllerEventThread) java.lang.NullPointerException at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:857) at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2559) at kafka.utils.Json$.parseBytes(Json.scala:62) at kafka.zk.ControllerZNode$.decode(ZkData.scala:56) at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:902) at kafka.controller.KafkaController.kafka$controller$KafkaController$elect(KafkaController.scala:1199) at kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1148) at kafka.controller.ControllerEventManager$ControllerEventThread$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:86) at kafka.controller.ControllerEventManager$ControllerEventThread$anonfun$doWork$1.apply(ControllerEventManager.scala:86) at kafka.controller.ControllerEventManager$ControllerEventThread$anonfun$doWork$1.apply(ControllerEventManager.scala:86) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31) at kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:85) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

2 REPLIES 2

Re: Apache 2.0 Kafka from HDP 3.1 Not Producing messages

Super Guru

NiFi 1.8.0 Publish Kafka Record 2 has no error and seems to send messages but nothing shows up

[root@princeton0 kafka]# /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list princeton0.field.hortonworks.com:6667 --topic bme680 >test [2018-12-23 17:17:18,101] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,190] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,293] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,398] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,501] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,603] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,708] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 7 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,812] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 8 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:18,917] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 9 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,022] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 10 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,126] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 11 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,230] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 12 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,334] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 13 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,443] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 14 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,547] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 15 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,650] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 16 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,754] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 17 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,858] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 18 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:19,962] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 19 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,066] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 20 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,169] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 21 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,273] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 22 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,377] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 23 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,480] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 24 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,584] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 25 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,689] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 26 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,792] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 27 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:20,896] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 28 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:21,003] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 29 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:21,106] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 30 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) [2018-12-23 17:17:21,209] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 31 : {bme680=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) ^Corg.apache.kafka.common.KafkaException: Producer closed while send in progress at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:826) at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803) at kafka.tools.ConsoleProducer$.send(ConsoleProducer.scala:75) at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:57) at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala) Caused by: org.apache.kafka.common.KafkaException: Requested metadata update after close at org.apache.kafka.clients.Metadata.awaitUpdate(Metadata.java:188) at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:938) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:823) ... 4 more

Re: Apache 2.0 Kafka from HDP 3.1 Not Producing messages

Super Guru

[root@princeton0 kafka-broker]# grep -iR advertised.host.name * conf/connect-distributed.properties:#rest.advertised.host.name= config/connect-distributed.properties:#rest.advertised.host.name= grep: libs/zookeeper.jar: No such file or directory logs/server.log:advertised.host.name = null logs/server.log:advertised.host.name = null