question Re: Nifi consumer for Rabbitmq streams in Support Questions

Nifi consumer for Rabbitmq streams

shiva239 — Tue, 13 Aug 2024 18:58:29 GMT

Hi,

Currently we are using ConsumeAMQP processor to fetch messages from Rabbitmq queues.

There are some use cases in our system where Rabbitmq streams seem to be a good fit.

I have tested ConsumeAMQP to consume messages from Rabbitmq streams queue, but it doesn't work as expected. I believe the new functionality related to OFFSET management is not available in ConsumeAMQP processor.

Any insights on this would be helpful. Am I missing something? If functionality is not part of current processor and would like it to be enhanced, how should I request?

Thanks in advance.

Re: Nifi consumer for Rabbitmq streams

joseomjr — Sun, 03 Nov 2024 00:27:49 GMT

Kinda neat...didn't know RabbitMQ had introduced "stream" queues...I've always used Kafka for streaming in the past. However, reading up on it I don't think NiFi has the native Java library/processors to use RabbitMQ streams. However, the version of the amqp-client that ships with NiFi should be able to support connecting to a streams queue if set up correctly meaning it might be possible to create a custom Groovy processor.

https://www.rabbitmq.com/docs/streams

Re: Nifi consumer for Rabbitmq streams

MattWho — Mon, 04 Nov 2024 14:09:49 GMT

@shiva239

You can create an apache "NiFi" jira in the Apache community to highlight this new feature and request modification to existing processor to support it.

https://issues.apache.org/jira/

The more detail you provide in your jira the better chance someone might take this on in the community.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt