Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Any example for Nifi integration with Phoenix on a kerberized cluster

Any example for Nifi integration with Phoenix on a kerberized cluster

Rising Star

This will be used as a lookup table. I am looking for this integration as HDF 2.1.1 doesnt have a processor to get the value of a specific rowkey in hbase.

5 REPLIES 5
Highlighted

Re: Any example for Nifi integration with Phoenix on a kerberized cluster

The ExecuteSql Apache NiFi processor should be sufficient for you to read data from Phoenix.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExecuteSQL/

Highlighted

Re: Any example for Nifi integration with Phoenix on a kerberized cluster

Rising Star

Fails with below error in Nifi:

ExecuteSQL[id=9f743ca0-425b-16a5-acdb-64784f4e18db] Unable to execute SQL select query select * from TABLE1 due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Thu May 04 09:32:46 EDT 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68339: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=uXX,16020,1490984879149, seqNum=0
). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Thu May 04 09:32:46 EDT 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68339: row 'SYSTEM:CATALOG,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=XXX,16020,1490984879149, seqNum=0

JDBC string:

jdbc:phoenix:XXX:2181:nifi@USE.UCDP.NET:/etc/security/keytabs/nifi.headless.keytab


untitled.png
Highlighted

Re: Any example for Nifi integration with Phoenix on a kerberized cluster

SocketTimeoutExceptions mean that the client couldn't open or read from a network connection to a remote system. There might be a firewall from the client machine to the server, the RegionServer may be dropping the connection (due to incorrect authentication), or the RegionServer is erroring out on the connection.

Look at the HBase logs on that host, provide more information about the HBase setup, and use network analysis tools (e.g. telnet) to verify that the client can make a network connection to that server+port.

Highlighted

Re: Any example for Nifi integration with Phoenix on a kerberized cluster

Rising Star

I dont see any messages on Hbase regionserver logs when I initiate the request. All ports are open between HDF and HDP. Telnet to regionserver port 16020 works from nifi nodes. Hbase setup has 2 masters and 6 regionservers with kerberos and ranger enabled.

Highlighted

Re: Any example for Nifi integration with Phoenix on a kerberized cluster

Set DEBUG logging on HBase -- it's probably a misconfiguration on your side with NiFi. DEBUG logging on HBase would prove this. Did you include a copy of hbase-site.xml and core-site.xml when providing the JAR to NiFi? If you didn't, the client isn't going to try to use Kerberos auth; there are many articles already covering how to do this.

Don't have an account?
Coming from Hortonworks? Activate your account here