Apache nifi and kafka microservice

alexmarco — Fri, 16 Jul 2021 09:05:38 GMT

I'm quite new in Apache Nifi but I'm encountering some issues trying to connect a kafka microservice (with a producer) with Apache nifi consumer.

Basically, I have a docker-compose like this:

zookeeper:
  container_name: zookeeper_test
  image: wurstmeister/zookeeper #zookeeper:3.5.7
  ports:
  - 2181:2181

kafka:
  container_name: kafka_test
  image: wurstmeister/kafka #:2.13-2.6.0
  ports:
  - 9092:9092
  environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  depends_on: 
      - zookeeper
      - kafkaui

kafkaui:
  container_name: kafka-ui_test
  image: provectuslabs/kafka-ui:latest
  environment: 
      - KAFKA_CLUSTERS_0_NAME=kafka
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER= zookeeper:2181
  ports:
      - 6789:8080

test:
  container_name: test
  build:
      context: ./test
      dockerfile: Dockerfile
  depends_on:
      - kafka
  command: python test.py

test is my producer:

from kafka import KafkaProducer import json from time import sleep producer = KafkaProducer(bootstrap_servers='kafka:9092') json_message = {"hello":"world"} for i in range(1000): producer.send("INPUT", json.dumps(json_message).encode('utf-8')) producer.flush() sleep(1)

Through the KafkaUI I'm able to see the topic SL.CPTI.INPUT that has been sent.

In the Apache nifi dashboard, I set a ConsumerKafka_2.6 with these parameters: Kafka broker: localhost:9092 Group ID: 1 Topic: SL.CPTI.INPUT

then I connected to this a funnel when "success", just to see the message received. Unfortunately, doing this, I do not see anything received. I just see a lot of tasks in the consumerkafka box but no elements in the queue connected to the funnel. I expect to see the json received, isn'it? May I miss something?

Re: Apache nifi and kafka microservice

alexmarco — Fri, 16 Jul 2021 12:42:50 GMT

it seems that the problem is related to the connection with kafka docker. In this moment I solved dockerize the NiFi installation, calling kafka as kafka:9092 from the apache nifi consumer

Re: Apache nifi and kafka microservice

cjervis — Fri, 16 Jul 2021 12:52:18 GMT

@alexmarco it seems like you resolved your issue. If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

question Re: Apache nifi and kafka microservice in Support Questions

Apache nifi and kafka microservice

Re: Apache nifi and kafka microservice

Re: Apache nifi and kafka microservice