HBase lookup via NiFi

[#1]

Hello all!

I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere?

I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!!

Thanks.

Valentin

[#2 · Accepted solution]

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven

[#3]

Thanks for the example @stevenmatison

I ended up using the LookupRecord with the HBase_1_1_2_RecordLookupService which seem to work.

Would you recommend one method over another for looking up records in HBase?

Valentin

[#4]

I recommend whatever works.  One of things i like the most about nifi is there are many ways to do similar things.  So if one way works for you, use it until you have some reason to try another way.

[#5]

Thanks @stevenmatison 

Do you by chance know the answer to this question https://community.cloudera.com/t5/Support-Questions/Extract-string-nested-in-JSON-value/m-p/305099

It's probably something very easy, but nothing that I tried works.

Valentin