Support Questions

Find answers, ask questions, and share your expertise

Nifi processors(publish and cosume kafka) are not communicating with kafka on Ambari platform!

91692-screenshot-from-2018-10-12-12-49-55.png

91693-screenshot-from-2018-10-12-12-48-54.png

I got stuck into this problem, nothing worked for me.
I want to establish a communication channel between nifi and kafka.

1-Four processors in nifi(GenerateFlowFile->publishKafka) and (ConsumeKafka->logAttribute).
2-PublishKafka and ConsumeKafka are not communicating with Apache kafka(pictures are attached along with their errors).
3-Apache kafka security=PLAINTEXT
4-Kebros not installed.

Please do suggest me the solution.
Thanks in advance.

1 REPLY 1

Mentor

@Muhammad waqas

In reality you don't communication channel between NiFi and Kafka. It's your PublishKafka and ConsumeKafka processors that connect behind the scenes!!!

First I see your processors are not started. Can you start them in this sequences

  • GenerateFlowFile
  • PublishKafka
  • LogAttribute
  • ConsumeKafka

That flow files should work you need to do the following

Create a Kafka topic Need to create a Kafka topic eg

./bin/kafka-topics.sh kafka-topic --create --topic Mytest --zookeeper 127.0.0.1:2181 --partitions 3 replication-factor 1 

In the PublishKafka configure the processor

Property value

Kafka brokers                127.0.0.1:9092 
Security protocol            PLAINTEXT                         
Topic Name                    mytest 
Delivery Guarantee           Guarantee Replicated Delivery          
  • Start the GeneraFlowFile processor
  • Start the PublishKafka processor

Configure and Start a kafka consumer

./bin/kafka-console-consumer.sh console-consumer --topic Mytest bootstrap-server 127.0.0.1:9092 

log attribute settings

  • Check Automatically Terminate Relationships

Apply and start the processor

Configure and Start a Kafka consumer

In the properties tab

Property Value

Kafka Brokers                   127.0.0.1:9092 
Security Protocol               PLAINTEXZT Topic Name                                        
Mytest Group ID                 Test (can be anything)Offset 
Reset                           latest or earliest 

Save by clicking apply

Now all your processor should show sgreen et voila !!