Support Questions
Find answers, ask questions, and share your expertise

Issues with DataSift streaming HTTP API working

Contributor

1309-screenshot-from-2016-01-13-11-01-24.png

I am trying to use InvokeHttp processor for the follwoing datasift https url. SSlcontext is now been setup. In the properties of InvokeHttp i gave https url and gave basic username and password which i was using for curl. I gave Auth in basic username and password. It starts up but doesnt pull any data. How do we stop it apart from restarting nifi. Also once the connection is made i want to keep it alive and doesn want it to get time out like in gethttp..??

. https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958 -H 'Auth: suri:dba37513923299cbb5bcbff766bacd3d

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Issues with DataSift streaming HTTP API working

Contributor

@surender nath reddy kudumula There are several other processors that handle various types of data formats and protocols and always more in development as we continue to grow the product and community. Some common items include UDP and Syslog listeners. I think there are certainly some additional items that we can add into NiFi to facilitate a use case such as streaming HTTP and actually started creating a ticket to provide some functionalities to at least cover general cases (NIFI-1389) Feel free to add any suggestions toward that functionality. Thanks!

View solution in original post

5 REPLIES 5

Re: Issues with DataSift streaming HTTP API working

Contributor

Surender:

The default NiFi does not currently have any processors that map directly to streamed HTTP data sources.

As a means of getting the data into a NiFi flow, you could also consider https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteProcess... processor.

Configuring an instance of this processor, with the properties:

CommandInfo

curl

Command Arguments

Info
-H "Auth: suri:dba37513923299cbb5bcbff766bacd3d" https://stream.datasift.com/fb409968ceacb8e588bb82de95c59958
Batch DurationInfo
1s

will give you provide a mechanism to bring this streamed content into the flow, batched into flowfiles at one second intervals. This gives a nice proof of concept of how you could interact with the data source before diving into a custom processor. However, this is inexact, and as a result, some results may get truncated depending on time boundaries. A custom processor might be nice to handle the data format to be cognizant of event boundaries.

Re: Issues with DataSift streaming HTTP API working

Contributor

@Aldrin Piri Interesting i thought we would have an out of the box processor as In internet of things where nifi is best suited i think we need an out of the box processor. SO if we are trying to stream well logs or sensor data using nifi which processors do we use? We have GetTwitter to stream tweets which is realtime.. Anyway thanks for the tips for using alternate processors for proof of concept. Thank you.

Re: Issues with DataSift streaming HTTP API working

Contributor

@surender nath reddy kudumula There are several other processors that handle various types of data formats and protocols and always more in development as we continue to grow the product and community. Some common items include UDP and Syslog listeners. I think there are certainly some additional items that we can add into NiFi to facilitate a use case such as streaming HTTP and actually started creating a ticket to provide some functionalities to at least cover general cases (NIFI-1389) Feel free to add any suggestions toward that functionality. Thanks!

View solution in original post

Re: Issues with DataSift streaming HTTP API working

Contributor

thanks @Aldrin Piri. ExecuteProcess processor fetches data. But i ll look for creating a custom processor. Thanks for the help.

Re: Issues with DataSift streaming HTTP API working

@Aldrin Piri Appreciate on opening the Jira to handle this.