Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase lookup via NiFi

Solved Go to solution
Highlighted

HBase lookup via NiFi

Explorer

Hello all!

I have a NiFi flow that tails a log file with IP addresses, and reads them into a flow. What processors do I need to chain in order to match each of the IP addresses from that flow against an HBase table, and when there is a match against that HBase table, to create a JSON which I will forward elsewhere?

 

I've spent most of the last two days trying different processors to create the match against HBase but with little success. By searching around I figured that for matching I need to use PutDistributedMapCache followed by a FetchDistributedMapCache but I'm struggling with the implementation. Any help, or links to articles are highly appreciated!!

 

Thanks.

Valentin

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: HBase lookup via NiFi

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

4 REPLIES 4
Highlighted

Re: HBase lookup via NiFi

@vatodorov19   i always find it interesting when someone posts on a topic I was just working on.  In this case I was working on Map/Cache Use Case and struggled to get it working until I found a template:

 

https://github.com/ds-steven-matison/NiFi-Templates/blob/main/DistributedCache_Demo.xml

 

Install the template and get the controller services working,  you will need to add the DistributedMapCacheServer before you can get the client working.  With server and client controller services enabled, run the flow.   Pay very close attention to the content of the flowfile sent to PutDistributedMapCache.  Next look at the Cache Entry Identifier in the properties of PutDistributedMapCache.

 

Once you have the demo working and are familiar with the basic implementaiton, modify this to fit your use case and operate within your flow.  If you have more specific questions about your flow, or getting the demo working please let me know.

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

 

Thanks,

Steven

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Highlighted

Re: HBase lookup via NiFi

Explorer

Thanks for the example @stevenmatison

I ended up using the LookupRecord with the HBase_1_1_2_RecordLookupService which seem to work.

Would you recommend one method over another for looking up records in HBase?

 

Valentin

Highlighted

Re: HBase lookup via NiFi

I recommend whatever works.  One of things i like the most about nifi is there are many ways to do similar things.  So if one way works for you, use it until you have some reason to try another way.

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Re: HBase lookup via NiFi

Explorer

Thanks @stevenmatison 

Do you by chance know the answer to this question https://community.cloudera.com/t5/Support-Questions/Extract-string-nested-in-JSON-value/m-p/305099

 

It's probably something very easy, but nothing that I tried works.

 

Valentin

Don't have an account?
Coming from Hortonworks? Activate your account here