Nifi GetHTTP more than one single processor

[#1]

Hello,

I want to make a stream for Yahoo weather.

But I do not want to use GetHTTP separately for each province. I would like to throw multiple queries in a single GetHTTP instead.

I would like to send the following 2 queries as an example

Query 1;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in...

Query 2;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in...

How do I submit them on a single processor?

there are 81 provinces like this. I do not want to do it all separately.

Thank you.

[#2 · Accepted solution]

@Murat Menteşe

Instead of GetHTTP processor use InvokeHTTP processor,InvokeHTTP Processor accepts incoming connections.

Use Generate Flowfile Processor and add all your URL's in GenerateFlowfile processors as each line having one URL in it.

Generate Flowfile Processor:-

Configs:-

As shown in the above screenshot add all URL's one per each line in CustomText property and In Scheduling tab change the frequency of run.

SplitText:-

We are Triggering GenerateFlowfile processor and we are going to have onefile having all url's in it so we are going to split the file into 1 Line as new flowfile now.

Configs:-

Line Split Count

1

THen connect Splits relation from SplitText processor to ExtractText processor.

ExtractText:-

We are going to Extract the contents of flowfile and adding as flowfile attribute in this processor then the extracted attribute is going to be used in InvokeHTTP processor.

Add new property in this processor by clicking on + sign at right corner

URL

(.*)

Configs:-

ControlRate:-

it's an optional processor as we are going to control the rate of flowfiles that are going to InvokeHTTP processor.

Rate Control Criteria

flowfile count

Maximum Rate

1

Rate Controlled Attribute No value set Time Duration

1 min

so with this configs we are going to process one flowfile per 1 min but you can change all this configs as per your requirements.

InvokeHTTP:-

we have configured RemoteURL property as ${URL} , this URL attribute we have added in ExtractText Processor.

So we don't need to use Seperate GetHTTP processors for each Request by following above approach.

Flow:-

GenerateFLowFIle //keep all URL's and schedule to run 
SplitText //Split into one line  
Extract Text //extracting URL from content of ff and adding as attribute 
ControlRate(optional) //control the rate of flowfiles passed to next processor
InvokeHTTP //Use the extracted URL attribute and Get Response. 

I'm attaching my xml here save, use as reference and modify as per your requirements.

hcc-168913-gethttp.xml

[#3]

thank you so much.

[#4]

Hello @Shu

what if i have all the URL's in a text file ?

like instead of opening generate flowfile and pasting all the urls manually , is there any way to send all the urls as a text file means without opening nifi console ?

by that we can do automate right ?

ManyThanks,

SaiKumar Akula.

[#5]

Hello,

I added the attachment file. With the GenerateFlowFile operator I sent the file to the SplitText operator. I've assigned it as a variable in the ExtractText operator. then I started to get the url diary about InvokeHTTP. @SaiKumar Akula

test-weather.xml

hope helps.

[#6]

@SaiKumar Akula

The same flow above will still work for you except swap out the GenerateFlowFile processor for a GetFile processor.

Simply set a "run schedule" for how often you want the GetFile to read the source file and change "Keep Source File" property to true.
Every time the processor executes it will read the source text file and it can be passed to the splitText processor in above flow and so on...

As long as you don't change the source file's filename, you can edit whenever you want to add new lines or remove URL lines. -

-

Thanks,

Matt