Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NDJSON stream with InvokeHTTP

NDJSON stream with InvokeHTTP

New Contributor

I have an API endpoint which sends back an NDJson response as an stream and without and end. I would need to send a single request and start parsing each of the Json files received, possibly insert them afterwards in a file. 


The issue here is that I don't know if InvokeHttp can "Open" a request and start receiving the info. Is this possible with this processor or is this something I would need to do in a different way.




Re: NDJSON stream with InvokeHTTP

Master Collaborator


Have you wired one up and seen what happens as a test?   I have recently used invokeHTTP to get streaming data from an amazon S3 bucket.  The requirement was to open a connection, then receive a constant stream of data.   Downstream from from InvokeHttp I am receiving files as expected without any out of the box configuration.


InvokeHttp should be able to handle what you need.  If you can report back your findings so that someone else looking for NDJSon context within the community has information.



If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  



Steven @ DFHZ


Re: NDJSON stream with InvokeHTTP

New Contributor

@stevenmatison I did a test configuring an InvokeHttp processor. I also made the same requests with Python and worked as intended. I'm getting the stream and sending this as producer in Kafka, because I need to create a topic so someone else can consume the info.


The test flow was created like this:

Main FlowMain Flow


also I've configured it with the URL as intended and added the bearer token as a dynamic property like this:



Although it doesn't show like there's an error, I'm not getting even a response code, and I guess I won't be receiving it as the file doesn't end at all and keeps receiving data.



Re: NDJSON stream with InvokeHTTP


Most web applications encounter problems of latency because they process data discretely instead of in streams. ndjsonstream() converts a ReadableStream of raw ndjson data into a ReadableStream of Javascript objects.

Don't have an account?
Coming from Hortonworks? Activate your account here