question Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey? in Support Questions

Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

anoop_nair — Tue, 19 Apr 2016 01:50:17 GMT

Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

sunile_manjee — Tue, 19 Apr 2016 01:54:44 GMT

@Anoop Nair Take a look at the getHbase processor here. Use filter on rowkey.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Tue, 19 Apr 2016 01:54:44 GMT

There is a GetHBase processor described here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.hbase.GetHBase/index.html

It is intended to do an incremental retrieval from an HBase table.

Does this work for your use-case, or are you looking for something different?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

anoop_nair — Tue, 19 Apr 2016 01:56:33 GMT

I'm not looking at a polling processor. The requirement is to query hbase on a specfic rowkey on demand basis.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

pvillard — Tue, 19 Apr 2016 02:23:13 GMT

What do you mean by on-demand basis?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

anoop_nair — Tue, 19 Apr 2016 04:46:41 GMT

The requirement is to read data from a kafka topic. The data has the rowkey that should be used to query data from a hbase table and send the column value to a JMS queue. Here is the flow:

Getkafka processor -> Gethbase -> Publish JMS

The problem is GetHbase processor cannot be an intermediate processor in the above chain.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Wed, 20 Apr 2016 01:30:38 GMT

I created this JIRA:

https://issues.apache.org/jira/browse/NIFI-1784

How would you want the data for the row to be represented in a NiFi flow file?

JSON document where the key/value pairs are column qualifiers / values?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Wed, 20 Apr 2016 01:37:43 GMT

Or are you getting the row id, col fam, col qual coming from kafka, and you want to fetch a single cell?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

anoop_nair — Wed, 20 Apr 2016 03:26:55 GMT

JSON document with key/value pairs should be fine. Just like the response from GetHbase processor. I may create a processor for now to get going.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Wed, 20 Apr 2016 21:27:41 GMT

@Anoop Nair if you do get something working we would be happy to help you contribute it back to NiFi if you would like to, although perfectly fine to maintain it outside of NiFi as well. Let us know if we can help.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

anoop_nair — Wed, 20 Apr 2016 21:55:59 GMT

@bbende I was thinking of extending nifi-hbase-bundle and adding a FetchHbaseProcessor using the existing capabilities in the bundle. I'm not sure if it's the right way. I'll give it a shot. Thanks for creating a JIRA request.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Wed, 20 Apr 2016 22:10:22 GMT

@Anoop Nair adding FetchHBase to the nifi-hbase-bundle sounds like the right approach.

The HBase processors rely on a Controller Service (HBaseClientService) to interact with HBase, which hides the HBase client from the processors. There is currently a scan method in the HBaseClientService:

public void scan(final String tableName, final Collection<Column> columns, final String filterExpression, final long minTime, final ResultHandler handler)
        throws IOException {

Technically you could probably use this with a filter expression limiting to the row id you want, but it would be a lot more efficient to create a new method in that interface that takes the table, row id, and columns to return.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

theenathayalan_ — Wed, 21 Dec 2016 12:34:00 GMT

@Anoop Nair Are you able to create gethasebykey? Appreciate if you could share your experience..We have similar use case

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Wed, 21 Dec 2016 21:55:08 GMT

@Theena R can you provide some input about your use-case...

Are you receiving messages from somewhere (Kafaka/JMS/etc) and those messages have a row key that you want to fetch?
Are you looking to fetch the whole row (i.e. multiple cells) or do you know a specific col-family:col-qualifier and only want to fetch a single cell?
What type of output would you expect? something similar to the JSON representation that GetHBase produces?

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

theenathayalan_ — Wed, 21 Dec 2016 22:24:59 GMT

@Bryan Bende Yes. On day 1, We are receiving JSON record as input which has a key and load into HBase using HbbasePut JSON. 2. On day 2, we get the same message with few changes. Need to determine if any changes and emit as delta to another processor. I am looking to query Hbase based on row key and get all other columns from Hbase and compare in the next processor (JSON path eval) and route as delta if there is any change in the values. GetHbase only scans a table and generates flowfile. I need to query on-demand based on a row query and get JSON representation as output.

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

bbende — Thu, 22 Dec 2016 01:36:21 GMT

@Theena R @Anoop Nair

I submitted a pull request for a FetchHBaseRow processor that may solve your use case:

https://github.com/apache/nifi/pull/1349

Keep in mind this is "use at your own risk" right now as it has not been reviewed or merged into the NiFi codebase and is subject to change at any time.

If you have any feedback, please comment on the JIRA:

https://issues.apache.org/jira/browse/NIFI-1784

Re: Does nifi (HDF 1.2) have a processor to read records from a hbase table based on a rowkey?

theenathayalan_ — Sat, 24 Dec 2016 03:06:37 GMT

@Bryan Bende Thank you. I am able to use it to fetch based on rowkey.