Support Questions
Find answers, ask questions, and share your expertise

HDF cluster Nifi Kafka consumer not working

Explorer

Hi Team,

We have configured HDF 3.1.1 single node instance for POC . He we are trying to perform a small test :

We have written a small python program to generate kafka message :

from kafka import KafkaProducer from kafka.errors import KafkaError producer = KafkaProducer(bootstrap_servers=' hostname:6667') topic = "jsontest" producer.send(topic, b'test message')

We want to consume this message thorough Nifi Consumekafka ,which is not working .

nifi.app.log

(type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:19,808 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:19,804 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:19,808; send took 4 millis 2018-05-15 07:51:20,352 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:21,505 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:22,357 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:23,560 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:24,557 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 8392 nanos 2018-05-15 07:51:24,668 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:24,811 INFO [Process Cluster Protocol Request-8] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 8aec83a3-cfbc-43c2-918d-143d701d13a8 (type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:24,815 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:24,809 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:24,815; send took 6 millis 2018-05-15 07:51:25,582 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:26,735 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:27,888 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:28,941 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:29,558 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in 7009 nanos 2018-05-15 07:51:29,817 INFO [Process Cluster Protocol Request-9] o.a.n.c.p.impl.SocketProtocolListener Finished processing request fcae4257-6a34-4691-9427-530ed061faf1 (type=HEARTBEAT, length=2086 bytes) from hdfdemo.c.corecompetetraining.internal:9090 in 1 millis 2018-05-15 07:51:29,819 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-05-15 07:51:29,815 and sent to hdfdemo.c.corecompetetraining.internal:9088 at 2018-05-15 07:51:29,819; send took 3 millis 2018-05-15 07:51:29,944 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:30,846 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available. 2018-05-15 07:51:31,999 WARN [Timer-Driven Process Thread-5] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-11, groupId=test] Connection to node -1 could not be established. Broker may not be available

nifi-user.log

2018-05-15 08:06:11,913 INFO [NiFi Web Server-387] org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) GET http://hostname:9090/nifi-api/flow/status (source ip: )

Please help on this .

Regards,

Vishal

3 REPLIES 3

Re: HDF cluster Nifi Kafka consumer not working

@vishal singh

Which ConsumeKafka processor are you using? Are you using the Kafka broker that is part of HDF-3.1.1?

Why not use the PublishKafka processor to post to the topic?

Re: HDF cluster Nifi Kafka consumer not working

Expert Contributor

Could you post the Kafka server.properties file? And could you post the output of 'ifconfig -a' also?

Re: HDF cluster Nifi Kafka consumer not working

@vishal singh

The error says that the Kafka Broker is not available.

Did you check if "python" is sending data to Kafka cluster without any error?

How did you setup ConsumeKafka Processor? Can you share a print screen?

This tutorial might be useful, please take a look

https://community.hortonworks.com/content/idea/191847/apache-nifi-apache-kafka-and-s3-tutorial.html