Whitelisting using LookupAttribute

[#1]

I am ingesting data from Kafka and then splitting out a JSON field. I then want to check that said fields content is in a whitelist that is kept in HDFS. I am looking at ScanAttribute and it seems like the thing I need except it can't read from HDFS. I tried reading from HDFS and then copying the file to local disk but that had some problems. Is there a better way?

• KafkaConsume
• SplitJsonPaths ($.someid)
• ScanAttribute from HDFS (${someid})
  • Drop if ${someid} text isn't in HDFS file (new-line-delimited text file)
  • Succeed if ${someid} text is in HDFS file

Chad

[#2]

Hi there,

I have broken out the import piece now so it doesn't run in line which is fine for now. The problem I have now is that ScanAttribute doesn't support expression language so I can't give it a variable to scan for... any suggestions?

[#3]

Can you give an example please? I can not see where the problem is

[#4]

Here is what I am doing currently. Essentially based on a field from the EvaluateJsonPath, lets call it ${someid}, I need to match that variable to a whitelist. That whitelist exists in HDFS but it could be loaded to local file or some cache if needed. If there is a match then I want to HTTPPost if not then I will drop the message. The issue is that ScanAttribute will only accept a regular expression and not a variable using expression language ${someid}.

I'm wondering if there is a better way. Maybe read in the whitelist to a CSV service and then lookup record... not sure how best to use a whitelist in Nifi.

[#5]

I ended up getting my whitelist to work. Here is what I did:

1. Put whitelist into csv format
2. Create a SimpleCSVLookupService
3. Use LookupAttribute to pull value of ${someid}. Returns null if nothing matched
4. Use RouteAttribute to test whither ${someid} equal the LookupAttribute

There might be a better way where I don't have to use RouteAttribute to compare two variables but this is working for me as of now.

[#6]

@Abdelkrim Hadjidj Your article was helpful as well:
https://community.hortonworks.com/articles/140231/data-flow-enrichment-with-nifi-part-2-lookupattrib...

[#7]

@Chad Shaw glad that you find the article useful 🙂