Unable to consume message from kafka broker in nifi

hegdemahendra — Wed, 19 Jul 2023 04:25:02 GMT

Hello experts,

We have setup our nifi to consume message from kafka broker which has SSL as security protocol.

It worked well with pre-prod brokers, but facing below error in prod broker.

(we have configured ssl context service with respective ssl certs)

Preprod nifi <-> Preprod kafka : working

Prod nifi <-> Prod kafka : Not working

telnet is working from Prod nifi to Prod kafka broker.

2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Connection to node -3 (kafka3.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.150:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
2023-07-17 15:30:58,814 WARN [Timer-Driven Process Thread-190] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdOnHandR3NiFiConsumer-28, groupId=ProdOnHandR3NiFiConsumer] Bootstrap broker kafka3.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -3 rack: null) disconnected
2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Connection to node -8 (kafka8.xxx.pivotal.emea.kaas.3stripes.net/xx.xxx.5.199:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
2023-07-17 15:30:59,816 WARN [Timer-Driven Process Thread-104] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmSeasonNiFiConsumer-26, groupId=ProdSpmSeasonNiFiConsumer] Bootstrap broker kafka8.xxx.pivotal.emea.kaas.3stripes.net:9093 (id: -8 rack: null) disconnected
2023-07-17 15:31:00,819 WARN [Timer-Driven Process Thread-127] org.apache.kafka.clients.NetworkClient [Consumer clientId=consumer-ProdSpmModelNiFiConsumer-25, groupId=ProdSpmModelNiFiConsumer] Connection to node -6 (kafka6.xxx.pivotal.emea.kaas.3stripes.net/xxx.128.xx.13:9093) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

Any idea what could be the reason for this?

Thanks

Mahendra

Re: Unable to consume message from kafka broker in nifi

Babasaheb — Wed, 06 Mar 2024 10:44:25 GMT

Hello @hegdemahendra,

1) Please refer to the following article to connect Kafka from Nifi:

https://community.cloudera.com/t5/Community-Articles/Integrating-Apache-NiFi-and-Apache-Kafka/ta-p/247433

2) Also, to isolate the issue you can try to connect Kafka from the same settings from nifi node using the Kafka command

Please let us know if you still have any questions regarding the same or facing any issues. We will be happy to assist you with it.

If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

question Re: Unable to consume message from kafka broker in nifi in Support Questions

Unable to consume message from kafka broker in nifi

Re: Unable to consume message from kafka broker in nifi