Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi & data-caching

Highlighted

Nifi & data-caching

New Contributor

Hi,

I am trying to make in-depth analysis of data-caching in Nifi. I have a sort of open questions about data-caching that came to my mind. Firstly, consider following scenario:

  • As input we get text file and we wish to save to location, which depends on it's filename (attribute) or content. In database, we may store information where given file should be stored (f.e file starting with 'abc' -> folder number 1 - simple table). In order to improve this process I want to store those information in cache, so that incoming file will look up for place to store first in state-store (cache) and then - if not-found - in database. The only solution that came to my mind is using Wait to store temporarily file till it's location is not determined - as shown in attachment photo. Will this be efficient enough? Can we process multiple files using this approach? Are there other ways to solve this problem? Generally, is nifi right option for this use-case?

Secondly, as far as I understand only 1MB state-store is available, which is used in many processors:

  • Is it possible to view it's whole content from GUI interface or any processor?

Thirdly, there are also group variables, which may be used as attributes in processors:

  • Can they be modified or created from processor level (UpdateAttribute)?

Finally, is there any other topic that I'm missing while exploring data-caching in Nifi?


przechwytywanie.png
Don't have an account?
Coming from Hortonworks? Activate your account here