Skip to main content
vatodorov19
vatodorov19
Silver ⭐️⭐️⭐️
October 26, 2020
Solved

HBase lookup via NiFi

Hello all!

I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere?

 

I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!!

 

Thanks.

Valentin

stevenmatison
stevenmatisonAnswer
Platinum ⭐️⭐️
October 28, 2020

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

View in thread

4 replies

stevenmatison
stevenmatisonAnswer
Platinum ⭐️⭐️
October 28, 2020

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

Translate
vatodorov19
vatodorov19Author
Silver ⭐️⭐️⭐️
October 29, 2020

Thanks for the example @stevenmatison

I ended up using the LookupRecord with the HBase_1_1_2_RecordLookupService which seem to work.

Would you recommend one method over another for looking up records in HBase?

 

Valentin

Translate
    stevenmatison
    stevenmatison
    Platinum ⭐️⭐️
    October 29, 2020

    I recommend whatever works.  One of things i like the most about nifi is there are many ways to do similar things.  So if one way works for you, use it until you have some reason to try another way.

    Translate
    Terms & ConditionsAccessibility statement