Support Questions
Find answers, ask questions, and share your expertise

NDJSON stream with InvokeHTTP

New Contributor

I have an API endpoint which sends back an NDJson response as an stream and without and end. I would need to send a single request and start parsing each of the Json files received, possibly insert them afterwards in a file. 

 

The issue here is that I don't know if InvokeHttp can "Open" a request and start receiving the info. Is this possible with this processor or is this something I would need to do in a different way.

 

Thanks!

3 REPLIES 3

@JesusRincon 

Have you wired one up and seen what happens as a test?   I have recently used invokeHTTP to get streaming data from an amazon S3 bucket.  The requirement was to open a connection, then receive a constant stream of data.   Downstream from from InvokeHttp I am receiving files as expected without any out of the box configuration.

 

InvokeHttp should be able to handle what you need.  If you can report back your findings so that someone else looking for NDJSon context within the community has information.

New Contributor

@stevenmatison I did a test configuring an InvokeHttp processor. I also made the same requests with Python and worked as intended. I'm getting the stream and sending this as producer in Kafka, because I need to create a topic so someone else can consume the info.

 

The test flow was created like this:

Main FlowMain Flow

 

also I've configured it with the URL as intended and added the bearer token as a dynamic property like this:

 

ProcessorConfig.PNG

Although it doesn't show like there's an error, I'm not getting even a response code, and I guess I won't be receiving it as the file doesn't end at all and keeps receiving data.

 

Explorer

Most web applications encounter problems of latency because they process data discretely instead of in streams. ndjsonstream() converts a ReadableStream of raw ndjson data into a ReadableStream of Javascript objects.