Support Questions
Find answers, ask questions, and share your expertise

Nifi GetHTTP more than one single processor

Solved Go to solution
Highlighted

Nifi GetHTTP more than one single processor

Hello,

I want to make a stream for Yahoo weather.

But I do not want to use GetHTTP separately for each province. I would like to throw multiple queries in a single GetHTTP instead.

I would like to send the following 2 queries as an example

Query 1;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in...

58428-adana.jpg

Query 2;

http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in...

58429-ankara.jpg

How do I submit them on a single processor?

there are 81 provinces like this. I do not want to do it all separately.

Thank you.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Nifi GetHTTP more than one single processor

Super Guru

@Murat Menteşe

Instead of GetHTTP processor use InvokeHTTP processor,InvokeHTTP Processor accepts incoming connections.

Use Generate Flowfile Processor and add all your URL's in GenerateFlowfile processors as each line having one URL in it.

Generate Flowfile Processor:-

Configs:-

58430-generateflowfile.png

As shown in the above screenshot add all URL's one per each line in CustomText property and In Scheduling tab change the frequency of run.

SplitText:-

We are Triggering GenerateFlowfile processor and we are going to have onefile having all url's in it so we are going to split the file into 1 Line as new flowfile now.

Configs:-

Line Split Count

1

THen connect Splits relation from SplitText processor to ExtractText processor.

ExtractText:-

We are going to Extract the contents of flowfile and adding as flowfile attribute in this processor then the extracted attribute is going to be used in InvokeHTTP processor.

Add new property in this processor by clicking on + sign at right corner

URL

(.*)

Configs:-

58431-extracttext.png

ControlRate:-

it's an optional processor as we are going to control the rate of flowfiles that are going to InvokeHTTP processor.

Rate Control Criteria

flowfile count

Maximum Rate

1

Rate Controlled Attribute No value set Time Duration

1 min

so with this configs we are going to process one flowfile per 1 min but you can change all this configs as per your requirements.

InvokeHTTP:-

58432-invokehttp.png

we have configured RemoteURL property as ${URL} , this URL attribute we have added in ExtractText Processor.

So we don't need to use Seperate GetHTTP processors for each Request by following above approach.

Flow:-

58433-flow.png

GenerateFLowFIle //keep all URL's and schedule to run 
SplitText //Split into one line  
Extract Text //extracting URL from content of ff and adding as attribute 
ControlRate(optional) //control the rate of flowfiles passed to next processor
InvokeHTTP //Use the extracted URL attribute and Get Response. 

I'm attaching my xml here save, use as reference and modify as per your requirements.

hcc-168913-gethttp.xml

View solution in original post

5 REPLIES 5
Highlighted

Re: Nifi GetHTTP more than one single processor

Super Guru

@Murat Menteşe

Instead of GetHTTP processor use InvokeHTTP processor,InvokeHTTP Processor accepts incoming connections.

Use Generate Flowfile Processor and add all your URL's in GenerateFlowfile processors as each line having one URL in it.

Generate Flowfile Processor:-

Configs:-

58430-generateflowfile.png

As shown in the above screenshot add all URL's one per each line in CustomText property and In Scheduling tab change the frequency of run.

SplitText:-

We are Triggering GenerateFlowfile processor and we are going to have onefile having all url's in it so we are going to split the file into 1 Line as new flowfile now.

Configs:-

Line Split Count

1

THen connect Splits relation from SplitText processor to ExtractText processor.

ExtractText:-

We are going to Extract the contents of flowfile and adding as flowfile attribute in this processor then the extracted attribute is going to be used in InvokeHTTP processor.

Add new property in this processor by clicking on + sign at right corner

URL

(.*)

Configs:-

58431-extracttext.png

ControlRate:-

it's an optional processor as we are going to control the rate of flowfiles that are going to InvokeHTTP processor.

Rate Control Criteria

flowfile count

Maximum Rate

1

Rate Controlled Attribute No value set Time Duration

1 min

so with this configs we are going to process one flowfile per 1 min but you can change all this configs as per your requirements.

InvokeHTTP:-

58432-invokehttp.png

we have configured RemoteURL property as ${URL} , this URL attribute we have added in ExtractText Processor.

So we don't need to use Seperate GetHTTP processors for each Request by following above approach.

Flow:-

58433-flow.png

GenerateFLowFIle //keep all URL's and schedule to run 
SplitText //Split into one line  
Extract Text //extracting URL from content of ff and adding as attribute 
ControlRate(optional) //control the rate of flowfiles passed to next processor
InvokeHTTP //Use the extracted URL attribute and Get Response. 

I'm attaching my xml here save, use as reference and modify as per your requirements.

hcc-168913-gethttp.xml

View solution in original post

Highlighted

Re: Nifi GetHTTP more than one single processor

thank you so much.

Highlighted

Re: Nifi GetHTTP more than one single processor

New Contributor

Hello @Shu

what if i have all the URL's in a text file ?

like instead of opening generate flowfile and pasting all the urls manually , is there any way to send all the urls as a text file means without opening nifi console ?

by that we can do automate right ?

ManyThanks,

SaiKumar Akula.

Re: Nifi GetHTTP more than one single processor

Hello,

I added the attachment file. With the GenerateFlowFile operator I sent the file to the SplitText operator. I've assigned it as a variable in the ExtractText operator. then I started to get the url diary about InvokeHTTP. @SaiKumar Akula

test-weather.xml

hope helps.

Highlighted

Re: Nifi GetHTTP more than one single processor

Master Guru

@SaiKumar Akula

The same flow above will still work for you except swap out the GenerateFlowFile processor for a GetFile processor.

Simply set a "run schedule" for how often you want the GetFile to read the source file and change "Keep Source File" property to true.
Every time the processor executes it will read the source text file and it can be passed to the splitText processor in above flow and so on...

As long as you don't change the source file's filename, you can edit whenever you want to add new lines or remove URL lines. -

-

Thanks,

Matt