Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Best way to generate dynamic URL in Nifi

Highlighted

Best way to generate dynamic URL in Nifi

New Contributor

I am using Nifi to generate dynamic URL to fetch the files automatically from website which is getting uploaded weekly. For example, The URL format seems like

https://XXX_030220_030820_YYY.zip (1st week) https://XXX_030920_031520_YYY.zip (2nd week)and so on....

and this format continues for further weeks with dynamic numbers representing the starting day(MMddyy) and ending day(MMddyy) of the respective week.

 

The solution which I did is using processors like

  1. GetFile - which get the text file where the URL is already pasted

  2. ExtractText - to save the URL in a property

  3. UpdateAttribute - used nifi expression language with substringAfter(XXX) and substringBefore(YYY) to get the numbers with underscore.Then used getDelimitedField to save the two numbers in two seperate properties by eliminating underscore.And then added seven days to both the property using toDate and toNumeber which exactly gives the next week dates.Then I replaced the two property values in https://XXX_{property1}_{property2}_YYY.zip and saved this link as attribute.

  4. InvokeHttp - which gets the file uploaded last week

  5. PutFile - to put the newly generated URL in new text file and gives the path mentioned in GetFile processor so that for next iteration getFile will take this text file and process goes on again to generate the next week url

This process works fine but seems like not a best solution. Could anyone please help me to give the best possible solution for my use case?

Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here