Issues with DataSift streaming HTTP API working

[#1]

I am trying to use InvokeHttp processor for the follwoing datasift https url. SSlcontext is now been setup. In the properties of InvokeHttp i gave https url and gave basic username and password which i was using for curl. I gave Auth in basic username and password. It starts up but doesnt pull any data. How do we stop it apart from restarting nifi. Also once the connection is made i want to keep it alive and doesn want it to get time out like in gethttp..??

. https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958 -H 'Auth: suri:dba37513923299cbb5bcbff766bacd3d

[#2]

Surender:

The default NiFi does not currently have any processors that map directly to streamed HTTP data sources.

As a means of getting the data into a NiFi flow, you could also consider https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteProcess... processor.

Configuring an instance of this processor, with the properties:

Command

curl

Command Arguments

-H "Auth: suri:dba37513923299cbb5bcbff766bacd3d" https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958

Batch Duration

1s

will give you provide a mechanism to bring this streamed content into the flow, batched into flowfiles at one second intervals. This gives a nice proof of concept of how you could interact with the data source before diving into a custom processor. However, this is inexact, and as a result, some results may get truncated depending on time boundaries. A custom processor might be nice to handle the data format to be cognizant of event boundaries.

[#3]

@Aldrin Piri Interesting i thought we would have an out of the box processor as In internet of things where nifi is best suited i think we need an out of the box processor. SO if we are trying to stream well logs or sensor data using nifi which processors do we use? We have GetTwitter to stream tweets which is realtime.. Anyway thanks for the tips for using alternate processors for proof of concept. Thank you.

[#4 · Accepted solution]

@surender nath reddy kudumula There are several other processors that handle various types of data formats and protocols and always more in development as we continue to grow the product and community. Some common items include UDP and Syslog listeners. I think there are certainly some additional items that we can add into NiFi to facilitate a use case such as streaming HTTP and actually started creating a ticket to provide some functionalities to at least cover general cases (NIFI-1389) Feel free to add any suggestions toward that functionality. Thanks!

[#5]

thanks @Aldrin Piri. ExecuteProcess processor fetches data. But i ll look for creating a custom processor. Thanks for the help.

[#6]

@Aldrin Piri Appreciate on opening the Jira to handle this.