Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while using publish kafka

Highlighted

Error while using publish kafka

Hello,

Trying to push message to kafka using PublishKafka processor.

Getting Error as given below:

2018-08-06 13:55:42,563 ERROR [Timer-Driven Process Thread-4] o.a.n.p.kafka.pubsub.PublishKafka_0_10 PublishKafka_0_10[id=01641014-a4dd-1982-315b-470a1ec73b5b] PublishKafka_0_10[id=01641014-a4dd-1982-315b-470a1ec73b5b] failed to process session due to org.apache.kafka.common.KafkaException: Failed to construct kafka producer: {} org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:342) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:166) at org.apache.nifi.processors.kafka.pubsub.PublisherPool.createLease(PublisherPool.java:61) at org.apache.nifi.processors.kafka.pubsub.PublisherPool.obtainPublisher(PublisherPool.java:56) at org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10.onTrigger(PublishKafka_0_10.java:312) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: ICMP Port Unreachable at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:51) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:305) ... 16 common frames omitted

2 REPLIES 2
Highlighted

Re: Error while using publish kafka

@Gitanjali Bare - Please confirm you have the correct settings in the Nifi Processor PublishKafka.

Here is a working example:

85552-publishkafka.jpg

Important settings:

  1. Kafka Broker: hostname:6667 - many times documentation shows a different port, use 6667
  2. Topic Name: Mine is a variable, you can enter anything here: "test", etc
  3. Kafka Key: Not required you can leave it empty.

If this answer helps, please click ACCEPT.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Re: Error while using publish kafka

@Steven Matison

Hi Steven,

Thanks For Reply.

1. Kafka Broker: hostname:6667 - many times documentation shows a different port, use 6667=== Used same port 6667

  • 2. Topic Name: Mine is a variable, you can enter anything here: "test", etc====Entered fixed topic
  • 3. Kafka Key: Not required you can leave it empty.====Its empty

    We are using Kerberized Kafka. Keytab and principal have defined in properties.

    PFA for properties.

    nifi-property.png

    Thanks,

    Gitanjali

  • Don't have an account?
    Coming from Hortonworks? Activate your account here