Whitelisting using LookupAttribute

[#1]

I am ingesting data from Kafka and then splitting out a JSON field. I then want to check that said fields content is in a whitelist that is kept in HDFS. I am looking at ScanAttribute and it seems like the thing I need except it can't read from HDFS. I tried reading from HDFS and then copying the file to local disk but that had some problems. Is there a better way?

• KafkaConsume
• SplitJsonPaths ($.someid)
• ScanAttribute from HDFS (${someid})
  • Drop if ${someid} text isn't in HDFS file (new-line-delimited text file)
  • Succeed if ${someid} text is in HDFS file

Chad

[#2 · Accepted solution]

Hi @Chad Shaw

How often your whitelist is updated? if not often, you can use NiFi to ingest it from HDFS, to store it in NiFi local server and use it with ScanAttribute.

[#3]

Hi there,

I have broken out the import piece now so it doesn't run in line which is fine for now. The problem I have now is that ScanAttribute doesn't support expression language so I can't give it a variable to scan for... any suggestions?

[#4]

Can you give an example please? I can not see where the problem is

[#5]

Here is what I am doing currently. Essentially based on a field from the EvaluateJsonPath, lets call it ${someid}, I need to match that variable to a whitelist. That whitelist exists in HDFS but it could be loaded to local file or some cache if needed. If there is a match then I want to HTTPPost if not then I will drop the message. The issue is that ScanAttribute will only accept a regular expression and not a variable using expression language ${someid}.

I'm wondering if there is a better way. Maybe read in the whitelist to a CSV service and then lookup record... not sure how best to use a whitelist in Nifi.

[#6]

I ended up getting my whitelist to work. Here is what I did:

1. Put whitelist into csv format
2. Create a SimpleCSVLookupService
3. Use LookupAttribute to pull value of ${someid}. Returns null if nothing matched
4. Use RouteAttribute to test whither ${someid} equal the LookupAttribute

There might be a better way where I don't have to use RouteAttribute to compare two variables but this is working for me as of now.

[#7]

@Abdelkrim Hadjidj Your article was helpful as well:
https://community.hortonworks.com/articles/140231/data-flow-enrichment-with-nifi-part-2-lookupattrib...

[#8]

@Chad Shaw glad that you find the article useful 🙂