Member since
02-03-2023
3
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2582 | 02-08-2023 02:51 PM |
02-06-2023
03:43 PM
Thanks for the reply! 1. My data flow is really small. I have 2 pipelines, that each consist of a consumeMQTT, encodeProtobuf and Publish. I am sending a single JSON message for it to go through the pipeline. 2. Is there a place to set that? I am running it as a Docker container on a 2019 6core I7. I have not limited the processors purposefully. I didn't see anything that would be, but that maybe something I am missing. Specifically using this setup: https://hub.docker.com/r/apache/nifi 3. I have tried increasing it slowly, I got it up to 100 without seeing any large change. I assume this is due to the single event being processed. What is a large value for this number? 4. I am not sure how to check this. 5. This does not seem outside of normal ranges if i am looking at the correct task. 6. The each pipeline is linear with 3 processors and only taking one input per test. This setup is just to verify the tool will work for our needs, so it is very simplistic. Which is why the processing time is confusing me 🙂 Thank you for your help!
... View more
02-03-2023
06:13 PM
I have been working with NiFi to do some basic testing with MQTT ingestion and publishing using protobufs. Everything seems to work fine, but each step takes about 30-60s to process. This seems incredibly long for such a simple process. I tried to see if there were any configuration settings that might need adjusted, but didn't have any success there. I also, setup a pipeline that consumes then publishes what it consumes and that still takes 30+ seconds. Is this a known issue or a setting that I have missed? I have used Streamsets in the past, so I decided to try it there and the same setup executes sub-second. I am using NiFi version 1.19.1 Appreciate any thoughts or assistance!
... View more
Labels:
- Labels:
-
Apache NiFi