I have an API endpoint which sends back an NDJson response as an stream and without and end. I would need to send a single request and start parsing each of the Json files received, possibly insert them afterwards in a file.
The issue here is that I don't know if InvokeHttp can "Open" a request and start receiving the info. Is this possible with this processor or is this something I would need to do in a different way.
Have you wired one up and seen what happens as a test? I have recently used invokeHTTP to get streaming data from an amazon S3 bucket. The requirement was to open a connection, then receive a constant stream of data. Downstream from from InvokeHttp I am receiving files as expected without any out of the box configuration.
InvokeHttp should be able to handle what you need. If you can report back your findings so that someone else looking for NDJSon context within the community has information.
If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.
Steven @ DFHZ
@stevenmatison I did a test configuring an InvokeHttp processor. I also made the same requests with Python and worked as intended. I'm getting the stream and sending this as producer in Kafka, because I need to create a topic so someone else can consume the info.
The test flow was created like this:
also I've configured it with the URL as intended and added the bearer token as a dynamic property like this:
Although it doesn't show like there's an error, I'm not getting even a response code, and I guess I won't be receiving it as the file doesn't end at all and keeps receiving data.