Support Questions
Find answers, ask questions, and share your expertise

Apache nifi and kafka microservice

Explorer

I'm quite new in Apache Nifi but I'm encountering some issues trying to connect a kafka microservice (with a producer) with Apache nifi consumer.

Basically, I have a docker-compose like this:

 

zookeeper:
  container_name: zookeeper_test
  image: wurstmeister/zookeeper #zookeeper:3.5.7
  ports:
  - 2181:2181

kafka:
  container_name: kafka_test
  image: wurstmeister/kafka #:2.13-2.6.0
  ports:
  - 9092:9092
  environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  depends_on: 
      - zookeeper
      - kafkaui

kafkaui:
  container_name: kafka-ui_test
  image: provectuslabs/kafka-ui:latest
  environment: 
      - KAFKA_CLUSTERS_0_NAME=kafka
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER= zookeeper:2181
  ports:
      - 6789:8080

test:
  container_name: test
  build:
      context: ./test
      dockerfile: Dockerfile
  depends_on:
      - kafka
  command: python test.py

test is my producer:

 

 

 

 

from kafka import KafkaProducer
import json
from time import sleep

producer = KafkaProducer(bootstrap_servers='kafka:9092')
json_message = {"hello":"world"}


for i in range(1000):
   producer.send("INPUT", json.dumps(json_message).encode('utf-8')) 
   producer.flush()
   sleep(1)

 

 

 

 

Through the KafkaUI I'm able to see the topic SL.CPTI.INPUT that has been sent.

In the Apache nifi dashboard, I set a ConsumerKafka_2.6 with these parameters: Kafka broker: localhost:9092 Group ID: 1 Topic: SL.CPTI.INPUT

then I connected to this a funnel when "success", just to see the message received. Unfortunately, doing this, I do not see anything received. I just see a lot of tasks in the consumerkafka box but no elements in the queue connected to the funnel. I expect to see the json received, isn'it? May I miss something?

 

Cattura1.PNGCattura2.PNGCattura3.PNG

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Apache nifi and kafka microservice

Explorer

it seems that the problem is related to the connection with kafka docker. In this moment I solved dockerize the NiFi installation, calling kafka as kafka:9092 from the apache nifi consumer

View solution in original post

2 REPLIES 2

Re: Apache nifi and kafka microservice

Explorer

it seems that the problem is related to the connection with kafka docker. In this moment I solved dockerize the NiFi installation, calling kafka as kafka:9092 from the apache nifi consumer

View solution in original post

Re: Apache nifi and kafka microservice

Community Manager

@alexmarco it seems like you resolved your issue. If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. 

Screen Shot 2019-08-06 at 1.54.47 PM.png

 

 


Cy Jervis, Manager, Community Program

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:
Community Guidelines
How to use the forum