HDP 2.5 - Zeppelin phoenix interpreter support in kerberos environment?

vishal423 — Tue, 15 Nov 2016 22:07:36 GMT

We have four node HDP 2.5 cluster and enabled kerberos on all services. During sanity check, we noticed that using Java client, we can connect to secured hbase using phoenix thick driver, however, via zeppelin phoenix interpreter (%phoenix) using same driver, I see below error in phoenix sample notebooks:

Failed after attempts=1, exceptions:
Tue Nov 15 13:50:01 UTC 2016, RpcRetryingCaller{globalStartTime=1479217801202, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipe
class org.apache.phoenix.exception.PhoenixIOException
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)

Following are phoenix specific interpreter settings:

phoenix.driver  -- org.apache.phoenix.jdbc.PhoenixDriver
phoenix.url  -- jdbc:phoenix:xxx:/hbase-secure
phoenix.user  -- phoenixuser
phoenix.password --
Dependencies -- org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1

If I change phoenix.url configuration to also specify hbase principal and keytab, then, I see below error:

phoenix.url  -- jdbc:phoenix:xxx:/hbase-secure:hbase@xxx:/etc/security/keytabs/hbase.headless.keytab

Log:

Failed after attempts=1, exceptions:
Tue Nov 15 13:25:09 UTC 2016, RpcRetryingCaller{globalStartTime=1479216309518, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to xxx/x.x.x.x:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to xxx/x.x.x.x:16000 is closing. Call id=0, waitTime=38
class org.apache.phoenix.exception.PhoenixIOException
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1063)
org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1369)
org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2116)
org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828)

Please note from above configurations and logs, I have purposefully masked host name and IP address.

I also noticed similar issue BUG-63165 at HDP 2.5 known issues, however, that seems to be more about impersonation. Any thoughts?

Re: HDP 2.5 - Zeppelin phoenix interpreter support in kerberos environment?

elserj — Wed, 16 Nov 2016 00:54:52 GMT

Using Zeppelin to run queries on your behalf is impersonation. It would appear that you are running into that exact known issue.

question HDP 2.5 - Zeppelin phoenix interpreter support in kerberos environment? in Archives of Support Questions (Read Only)

HDP 2.5 - Zeppelin phoenix interpreter support in kerberos environment?

Re: HDP 2.5 - Zeppelin phoenix interpreter support in kerberos environment?