Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Whitelisting using LookupAttribute

avatar
Explorer

I am ingesting data from Kafka and then splitting out a JSON field. I then want to check that said fields content is in a whitelist that is kept in HDFS. I am looking at ScanAttribute and it seems like the thing I need except it can't read from HDFS. I tried reading from HDFS and then copying the file to local disk but that had some problems. Is there a better way?

  • KafkaConsume
  • SplitJsonPaths ($.someid)
  • ScanAttribute from HDFS (${someid})
    • Drop if ${someid} text isn't in HDFS file (new-line-delimited text file)
    • Succeed if ${someid} text is in HDFS file

Chad

1 ACCEPTED SOLUTION

avatar

Hi @Chad Shaw

How often your whitelist is updated? if not often, you can use NiFi to ingest it from HDFS, to store it in NiFi local server and use it with ScanAttribute.

View solution in original post

7 REPLIES 7

avatar

Hi @Chad Shaw

How often your whitelist is updated? if not often, you can use NiFi to ingest it from HDFS, to store it in NiFi local server and use it with ScanAttribute.

avatar
Explorer

Hi there,

I have broken out the import piece now so it doesn't run in line which is fine for now. The problem I have now is that ScanAttribute doesn't support expression language so I can't give it a variable to scan for... any suggestions?

avatar

Can you give an example please? I can not see where the problem is

avatar
Explorer

72529-screen-shot-2018-05-01-at-101117-am.png

Here is what I am doing currently. Essentially based on a field from the EvaluateJsonPath, lets call it ${someid}, I need to match that variable to a whitelist. That whitelist exists in HDFS but it could be loaded to local file or some cache if needed. If there is a match then I want to HTTPPost if not then I will drop the message. The issue is that ScanAttribute will only accept a regular expression and not a variable using expression language ${someid}.

I'm wondering if there is a better way. Maybe read in the whitelist to a CSV service and then lookup record... not sure how best to use a whitelist in Nifi.

avatar
Explorer

I ended up getting my whitelist to work. Here is what I did:

  1. Put whitelist into csv format
  2. Create a SimpleCSVLookupService
  3. Use LookupAttribute to pull value of ${someid}. Returns null if nothing matched
  4. Use RouteAttribute to test whither ${someid} equal the LookupAttribute

There might be a better way where I don't have to use RouteAttribute to compare two variables but this is working for me as of now.

72531-screen-shot-2018-05-01-at-14829-pm.png


screen-shot-2018-05-01-at-14829-pm.png

avatar
Explorer

avatar

@Chad Shaw glad that you find the article useful 🙂