question Nifi GetHTTP more than one single processor in Archives of Support Questions (Read Only)

Nifi GetHTTP more than one single processor

muratmentese1 — Sun, 18 Aug 2019 04:49:26 GMT

Hello,

I want to make a stream for Yahoo weather.

But I do not want to use GetHTTP separately for each province. I would like to throw multiple queries in a single GetHTTP instead.

I would like to send the following 2 queries as an example

Query 1;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22Adana%2CAdana%2C%20tr%22)%20and%20u%3D%22c%22&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

Query 2;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22Ankara%2CAnkara%2C%20tr%22)%20and%20u%3D%22c%22&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

How do I submit them on a single processor?

there are 81 provinces like this. I do not want to do it all separately.

Thank you.

Re: Nifi GetHTTP more than one single processor

Shu_ashu — Sun, 18 Aug 2019 04:49:13 GMT

@Murat Menteşe

Instead of GetHTTP processor use InvokeHTTP processor,InvokeHTTP Processor accepts incoming connections.

Use Generate Flowfile Processor and add all your URL's in GenerateFlowfile processors as each line having one URL in it.

Generate Flowfile Processor:-

Configs:-

As shown in the above screenshot add all URL's one per each line in CustomText property and In Scheduling tab change the frequency of run.

SplitText:-

We are Triggering GenerateFlowfile processor and we are going to have onefile having all url's in it so we are going to split the file into 1 Line as new flowfile now.

Configs:-

Line Split Count

THen connect Splits relation from SplitText processor to ExtractText processor.

ExtractText:-

We are going to Extract the contents of flowfile and adding as flowfile attribute in this processor then the extracted attribute is going to be used in InvokeHTTP processor.

Add new property in this processor by clicking on + sign at right corner

URL

(.*)

Configs:-

ControlRate:-

it's an optional processor as we are going to control the rate of flowfiles that are going to InvokeHTTP processor.

Rate Control Criteria

flowfile count

Maximum Rate

Rate Controlled Attribute No value set Time Duration

1 min

so with this configs we are going to process one flowfile per 1 min but you can change all this configs as per your requirements.

InvokeHTTP:-

we have configured RemoteURL property as ${URL} , this URL attribute we have added in ExtractText Processor.

So we don't need to use Seperate GetHTTP processors for each Request by following above approach.

Flow:-

GenerateFLowFIle //keep all URL's and schedule to run 
SplitText //Split into one line  
Extract Text //extracting URL from content of ff and adding as attribute 
ControlRate(optional) //control the rate of flowfiles passed to next processor
InvokeHTTP //Use the extracted URL attribute and Get Response.

I'm attaching my xml here save, use as reference and modify as per your requirements.

hcc-168913-gethttp.xml

Re: Nifi GetHTTP more than one single processor

muratmentese1 — Sun, 28 Jan 2018 04:58:19 GMT

thank you so much.

Re: Nifi GetHTTP more than one single processor

saikumar_akula2 — Tue, 17 Jul 2018 19:37:34 GMT

Hello @Shu

what if i have all the URL's in a text file ?

like instead of opening generate flowfile and pasting all the urls manually , is there any way to send all the urls as a text file means without opening nifi console ?

by that we can do automate right ?

ManyThanks,

SaiKumar Akula.

Re: Nifi GetHTTP more than one single processor

muratmentese1 — Tue, 17 Jul 2018 20:00:52 GMT

Hello,

I added the attachment file. With the GenerateFlowFile operator I sent the file to the SplitText operator. I've assigned it as a variable in the ExtractText operator. then I started to get the url diary about InvokeHTTP. @SaiKumar Akula

test-weather.xml

hope helps.

Re: Nifi GetHTTP more than one single processor

MattWho — Tue, 17 Jul 2018 20:42:31 GMT

@SaiKumar Akula

The same flow above will still work for you except swap out the GenerateFlowFile processor for a GetFile processor.

Simply set a "run schedule" for how often you want the GetFile to read the source file and change "Keep Source File" property to true.
Every time the processor executes it will read the source text file and it can be passed to the splitText processor in above flow and so on...

As long as you don't change the source file's filename, you can edit whenever you want to add new lines or remove URL lines. -

Thanks,

Matt