question Re: Issues with DataSift streaming HTTP API working in Archives of Support Questions (Read Only)

Issues with DataSift streaming HTTP API working

bigspark — Mon, 19 Aug 2019 12:15:44 GMT

I am trying to use InvokeHttp processor for the follwoing datasift https url. SSlcontext is now been setup. In the properties of InvokeHttp i gave https url and gave basic username and password which i was using for curl. I gave Auth in basic username and password. It starts up but doesnt pull any data. How do we stop it apart from restarting nifi. Also once the connection is made i want to keep it alive and doesn want it to get time out like in gethttp..??

. https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958 -H 'Auth: suri:dba37513923299cbb5bcbff766bacd3d

Re: Issues with DataSift streaming HTTP API working

apiri — Thu, 14 Jan 2016 06:54:17 GMT

Surender:

The default NiFi does not currently have any processors that map directly to streamed HTTP data sources.

As a means of getting the data into a NiFi flow, you could also consider https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteProcess/index.html processor.

Configuring an instance of this processor, with the properties:

Command

curl

Command Arguments

-H "Auth: suri:dba37513923299cbb5bcbff766bacd3d" https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958

Batch Duration

1s

will give you provide a mechanism to bring this streamed content into the flow, batched into flowfiles at one second intervals. This gives a nice proof of concept of how you could interact with the data source before diving into a custom processor. However, this is inexact, and as a result, some results may get truncated depending on time boundaries. A custom processor might be nice to handle the data format to be cognizant of event boundaries.

Re: Issues with DataSift streaming HTTP API working

bigspark — Thu, 14 Jan 2016 07:03:07 GMT

@Aldrin Piri Interesting i thought we would have an out of the box processor as In internet of things where nifi is best suited i think we need an out of the box processor. SO if we are trying to stream well logs or sensor data using nifi which processors do we use? We have GetTwitter to stream tweets which is realtime.. Anyway thanks for the tips for using alternate processors for proof of concept. Thank you.

Re: Issues with DataSift streaming HTTP API working

apiri — Thu, 14 Jan 2016 07:16:59 GMT

@surender nath reddy kudumula There are several other processors that handle various types of data formats and protocols and always more in development as we continue to grow the product and community. Some common items include UDP and Syslog listeners. I think there are certainly some additional items that we can add into NiFi to facilitate a use case such as streaming HTTP and actually started creating a ticket to provide some functionalities to at least cover general cases (NIFI-1389) Feel free to add any suggestions toward that functionality. Thanks!

Re: Issues with DataSift streaming HTTP API working

bigspark — Thu, 14 Jan 2016 07:42:45 GMT

thanks @Aldrin Piri. ExecuteProcess processor fetches data. But i ll look for creating a custom processor. Thanks for the help.

Re: Issues with DataSift streaming HTTP API working

nsabharwal — Sun, 17 Jan 2016 09:35:20 GMT

@Aldrin Piri Appreciate on opening the Jira to handle this.