Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HBase lookup via NiFi

avatar
Contributor

Hello all!

I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere?

 

I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!!

 

Thanks.

Valentin

1 ACCEPTED SOLUTION

avatar
Super Guru

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

View solution in original post

4 REPLIES 4

avatar
Super Guru

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

avatar
Contributor

Thanks for the example @stevenmatison

I ended up using the LookupRecord with the HBase_1_1_2_RecordLookupService which seem to work.

Would you recommend one method over another for looking up records in HBase?

 

Valentin

avatar
Super Guru

I recommend whatever works.  One of things i like the most about nifi is there are many ways to do similar things.  So if one way works for you, use it until you have some reason to try another way.

avatar
Contributor

Thanks @stevenmatison 

Do you by chance know the answer to this question https://community.cloudera.com/t5/Support-Questions/Extract-string-nested-in-JSON-value/m-p/305099

 

It's probably something very easy, but nothing that I tried works.

 

Valentin