Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi processor to fetch attribute value dynamically from file or table

Solved Go to solution
Highlighted

NiFi processor to fetch attribute value dynamically from file or table

Rising Star

I want processor to fetch attribute value on run time. Example - if I am filtering twitter feeds by specific keywords, i want to maintain the list of keywords in a separate repository like file or table and not confined as a text box value. In that case, how will NiFi processor fetch that values from external file or table to attribute value.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Rising Star

Thanks @Timothy Spann. Looked at your comments just now after i did some modifications to existing nar file. I edited the current ScanAttribute file to iterate the dictionary words and check for its presence in twitter message. Customizing is good and look forward to make more solutions integrated into NiFi.

View solution in original post

7 REPLIES 7
Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

It sounds like you might be able to use the ScanAttribute Processor to meet your needs.

Re: NiFi processor to fetch attribute value dynamically from file or table

Rising Star

Hi. Appreciate your answer. Sorry for delay in response - I was away for a while. ScanAttribute processor sounds to be the answer to my question and I did try a sample but its not picking up value and filtering the incoming tweets accordingly. Can you provide an example with snapshots.

Incoming tweets via getTwitter Processor >> ScanAttribute processor. (dictionary file : <location of text file with keywords), attribute pattern - $.text, Match criteria - atleast one value must match)

Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Rising Star

ScanAttribute processor works fine for matching exact value and not substring of attribute value. I looked into the code {{if (dictionary.contains(entry.getValue())) }} which means matching condition if dictionary words contain any of attribute words. In my case attribute is a tweet - collection of words and would fail all as unmatched.

Need to look at a logic to see if attribute : tweet (collection of words) contain dictionary words. Customization of processor or other ways of extract text from tweet attribute.

Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Rising Star

@Timothy Spann : Greetings. I need your assistance in customizing processor - ScanAttribute.

Flow file data considering it as tweet_msg attribute needs to be checked if it contains dictionary file words.

Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Super Guru

Scan attribute should work just put in a regular expression that does what you need and just use the one attribute for . maybe it's updated in the new one.

You can create your own processor, starting with the code in ScanAttribute if it doesn't meet your needs then deploy your NAR to the nifi/lib dir and restart nifi.

This is the formatting

https://github.com/tspannhw/nifi-nlp-processor

copy this code

https://github.com/apache/nifi/blob/f8cad0f8c94869e2fd2a30896033c3c48eadbf14/nifi-nar-bundles/nifi-s...

https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-s...

Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Rising Star

Thanks @Timothy Spann. Looked at your comments just now after i did some modifications to existing nar file. I edited the current ScanAttribute file to iterate the dictionary words and check for its presence in twitter message. Customizing is good and look forward to make more solutions integrated into NiFi.

View solution in original post

Highlighted

Re: NiFi processor to fetch attribute value dynamically from file or table

Super Guru

cool, put a JIRA ticket out for that and post it to your own github. perhaps that will get mainlined. Accept your answer.

Don't have an account?
Coming from Hortonworks? Activate your account here