I was looking for a server sent event (SSE) client in Apache NiFi, however, I couldn't find any ready processor that can do that.
I started implementing an SSE client using Python script and used ExecuteCommand processor to run this script. However, the script needs to be terminated in order to send the processor output to the next step through STDOUT. (ie. I can't use an infinite Loop "While True:" to listen to the SSE server and output the consumed events on stream).
Is there any ideas to implement the SSE client in NiFi such that consumed events are processed one by one to the next processors in real-time?
@mmaher22 You may want to run the python job inside of ExecuteScript. In this manner, you can send output to a flowfile during your loops iterations with:
This command is inferred at the end of the code execution in ExecuteScript to send output to next processor (1 flow file). So if you just put that in line with your loop, then the script will run, and send flowfiles for every instance.
For a full rundown of how to use ExecuteScript be sure to see these great articles:
I spun my wheels on this for quite a while with no success; I can get the authorization token, but that's it. Do you have an example of using a script in the ExecuteScript (or ExecuteGroovyScript) that can make an HTTP request for a token and then use that token to start an SSE stream? I'd really appreciate whatever you are willing to share. Many thanks!
Here is what I've come up with so far, but I can't get the SSE responses to output to flowfiles.
Just to add to this, I created a Java version of this code which I verified works from the command line; I get the SSE feed printing to the console. However, when I attempt to use this same code in an ExecuteStreamCommand processor then I get the exact same behavior, which is that the processor is running but there isn't any data coming out of it. I'm missing a detail that I hope someone can shed some light on.