We have four node HDP 2.5 cluster and enabled kerberos on all services. During sanity check, we noticed that using Java client, we can connect to secured hbase using phoenix thick driver, however, via zeppelin phoenix interpreter (%phoenix) using same driver, I see below error in phoenix sample notebooks:
Failed after attempts=1, exceptions:
Tue Nov 15 13:50:01 UTC 2016, RpcRetryingCaller{globalStartTime=1479217801202, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.io.IOException: Broken pipe
class org.apache.phoenix.exception.PhoenixIOException
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)Following are phoenix specific interpreter settings:
phoenix.driver -- org.apache.phoenix.jdbc.PhoenixDriver
phoenix.url -- jdbc:phoenix:xxx:/hbase-secure
phoenix.user -- phoenixuser
phoenix.password --
Dependencies -- org.apache.phoenix:phoenix-core:4.7.0-HBase-1.1
If I change phoenix.url configuration to also specify hbase principal and keytab, then, I see below error:
phoenix.url -- jdbc:phoenix:xxx:/hbase-secure:hbase@xxx:/etc/security/keytabs/hbase.headless.keytab
Log:
Failed after attempts=1, exceptions:
Tue Nov 15 13:25:09 UTC 2016, RpcRetryingCaller{globalStartTime=1479216309518, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to xxx/x.x.x.x:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to xxx/x.x.x.x:16000 is closing. Call id=0, waitTime=38
class org.apache.phoenix.exception.PhoenixIOException
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1063)
org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1369)
org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2116)
org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:828)
Please note from above configurations and logs, I have purposefully masked host name and IP address.
I also noticed similar issue BUG-63165 at HDP 2.5 known issues, however, that seems to be more about impersonation. Any thoughts?
