Apache nifi and kafka microservice
I'm quite new in Apache Nifi but I'm encountering some issues trying to connect a kafka microservice (with a producer) with Apache nifi consumer.
Basically, I have a docker-compose like this:
zookeeper:
container_name: zookeeper_test
image: wurstmeister/zookeeper #zookeeper:3.5.7
ports:
- 2181:2181
kafka:
container_name: kafka_test
image: wurstmeister/kafka #:2.13-2.6.0
ports:
- 9092:9092
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
- kafkaui
kafkaui:
container_name: kafka-ui_test
image: provectuslabs/kafka-ui:latest
environment:
- KAFKA_CLUSTERS_0_NAME=kafka
- KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
- KAFKA_CLUSTERS_0_ZOOKEEPER= zookeeper:2181
ports:
- 6789:8080
test:
container_name: test
build:
context: ./test
dockerfile: Dockerfile
depends_on:
- kafka
command: python test.py
test is my producer:
from kafka import KafkaProducer
import json
from time import sleep
producer = KafkaProducer(bootstrap_servers='kafka:9092')
json_message = {"hello":"world"}
for i in range(1000):
producer.send("INPUT", json.dumps(json_message).encode('utf-8'))
producer.flush()
sleep(1)
Through the KafkaUI I'm able to see the topic SL.CPTI.INPUT that has been sent.
In the Apache nifi dashboard, I set a ConsumerKafka_2.6 with these parameters: Kafka broker: localhost:9092 Group ID: 1 Topic: SL.CPTI.INPUT
then I connected to this a funnel when "success", just to see the message received. Unfortunately, doing this, I do not see anything received. I just see a lot of tasks in the consumerkafka box but no elements in the queue connected to the funnel. I expect to see the json received, isn'it? May I miss something?
