Skip to main content
Shalexey
Gold ⭐️
October 22, 2024
Solved

Nifi - filter lookup table

Hi, experts,

I need to enrich flowfile with data from database table, but before this table should be filtered based on attribute value. Is it possible to filter data within lookup processor? If not, please advise any solution for my task.

Best regards,
Alexey

Shalexey_0-1729596069514.png

 

 

Shalexey
ShalexeyAnswer
Gold ⭐️
October 25, 2024

Finally I replaced LookupRecord processor with group of Fork/JoinEnrichment.

Shalexey_0-1729860622172.png

 

3 replies

drewski7
Gold ⭐️⭐️
October 23, 2024

@Shalexey 

Can you provide an example of what you mean by filter?

The LookupRecord processor primary function is to extract one or more fields from a Record and looks up a value for those fields in a LookupService.

If you are trying to do a filter you can use another NiFi processor prior to the LookupRecord processor. Here are some examples below...

1. RouteOnAttribute

Routes flow files based on attribute values by evaluating expressions. It allows you to define conditions (e.g., ${status:equals('SUCCESS')}) to send data to different relationships, like success or unmatched.

2. RouteOnContent

Filters flow files based on patterns within the content. This processor is ideal for detecting specific keywords, tags, or values inside text, XML, or JSON data and routing them accordingly.

3. FilterCSV

Designed to filter rows in CSV files based on specific column values or positions. It removes unwanted data from the flow, only passing records that meet the defined criteria (e.g., status = 'active').

4. QueryRecord

Uses SQL queries to filter structured data (JSON, CSV, Avro) within flow files. You can select or discard records by writing SQL-like queries (e.g., SELECT * FROM FLOWFILE WHERE type = 'transaction').

 

Translate
Shalexey
ShalexeyAuthor
Gold ⭐️
October 25, 2024

I'm looking up for ID in a table with following data:

id,key,attribute
1,same_value,attr1
2,same_value,attr2
3,same_value,attr3

As "key" has several records I need to filter the table by "attribute" in order to get one correct "id".

Translate
Shalexey
ShalexeyAuthorAnswer
Gold ⭐️
October 25, 2024

Finally I replaced LookupRecord processor with group of Fork/JoinEnrichment.

Shalexey_0-1729860622172.png

 

Translate