question Whitelisting using LookupAttribute in Archives of Support Questions (Read Only)

Whitelisting using LookupAttribute

lbmission — Tue, 01 May 2018 10:22:37 GMT

I am ingesting data from Kafka and then splitting out a JSON field. I then want to check that said fields content is in a whitelist that is kept in HDFS. I am looking at ScanAttribute and it seems like the thing I need except it can't read from HDFS. I tried reading from HDFS and then copying the file to local disk but that had some problems. Is there a better way?

KafkaConsume
SplitJsonPaths ($.someid)
ScanAttribute from HDFS (${someid})
- Drop if ${someid} text isn't in HDFS file (new-line-delimited text file)
- Succeed if ${someid} text is in HDFS file

Chad

Re: Whitelisting using LookupAttribute

ahadjidj — Tue, 01 May 2018 19:28:20 GMT

Hi @Chad Shaw

How often your whitelist is updated? if not often, you can use NiFi to ingest it from HDFS, to store it in NiFi local server and use it with ScanAttribute.

Re: Whitelisting using LookupAttribute

lbmission — Tue, 01 May 2018 22:49:38 GMT

Hi there,

I have broken out the import piece now so it doesn't run in line which is fine for now. The problem I have now is that ScanAttribute doesn't support expression language so I can't give it a variable to scan for... any suggestions?

Re: Whitelisting using LookupAttribute

ahadjidj — Tue, 01 May 2018 22:52:13 GMT

Can you give an example please? I can not see where the problem is

Re: Whitelisting using LookupAttribute

lbmission — Sun, 18 Aug 2019 08:54:21 GMT

Here is what I am doing currently. Essentially based on a field from the EvaluateJsonPath, lets call it ${someid}, I need to match that variable to a whitelist. That whitelist exists in HDFS but it could be loaded to local file or some cache if needed. If there is a match then I want to HTTPPost if not then I will drop the message. The issue is that ScanAttribute will only accept a regular expression and not a variable using expression language ${someid}.

I'm wondering if there is a better way. Maybe read in the whitelist to a CSV service and then lookup record... not sure how best to use a whitelist in Nifi.

Re: Whitelisting using LookupAttribute

lbmission — Sun, 18 Aug 2019 08:54:14 GMT

I ended up getting my whitelist to work. Here is what I did:

Put whitelist into csv format
Create a SimpleCSVLookupService
Use LookupAttribute to pull value of ${someid}. Returns null if nothing matched
Use RouteAttribute to test whither ${someid} equal the LookupAttribute

There might be a better way where I don't have to use RouteAttribute to compare two variables but this is working for me as of now.

Re: Whitelisting using LookupAttribute

lbmission — Wed, 02 May 2018 03:21:28 GMT

@Abdelkrim Hadjidj Your article was helpful as well:
https://community.hortonworks.com/articles/140231/data-flow-enrichment-with-nifi-part-2-lookupattrib.html

Re: Whitelisting using LookupAttribute

ahadjidj — Wed, 02 May 2018 03:38:54 GMT

@Chad Shaw glad that you find the article useful 🙂