Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase-Indexer, Kerberos and HBase

HBase-Indexer, Kerberos and HBase

New Contributor

Hello,

I'm trying to setup hbase-indexer (NGDATA) in a HDP (2.5) kerberized environement.

- Solr and Hbase-Indexer are on the same machine (host15)

- HMaster is in a separate VM (host12)

- 2 Region servers in two other machine (host13 and host14)

- All the VMs are on the same domain

- I'm using a separate AD / Kerberos

- ZK Quorum is host12 + host13 + host14

I followed the HortonWorks tutorial to set it up, but this tutorial contains some "fuzzy" details such as the ALL_FQDN SPN in the jaas file.

Note that the kerberized cluster is working fine and I'm able to connect with my own code.

Now, when it comes to Hbase-indexer, I'm getting this exception (Kerberos principal name does NOT have the expected hostname part: hbase):

  2018-04-17 16:41:51,032 DEBUG [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/host15.secure.com@IBMDEV.LOCAL  
2018-04-17 16:41:51,033 DEBUG [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
security.HBaseSaslRpcClient: Have sent token of size 1472 from initSASLContext.  2018-04-17
16:41:51,033 DEBUG [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
ipc.AbstractRpcClient: Exception encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase  2018-04-17
16:41:55,780 DEBUG [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
principal name is hbase/host15.secure.com@IBMDEV.LOCAL  2018-04-17
16:41:55,781 DEBUG [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
security.HBaseSaslRpcClient: Have sent token of size 1473 from initSASLContext.  2018-04-17
16:41:55,781 WARN  [ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
ipc.AbstractRpcClient: Couldn't setup connection for hbase/host13.secure.com@IBMDEV.LOCAL
to hbase/host15.secure.com@IBMDEV.LOCAL  2018-04-17
16:41:55,781 WARN 
[ReplicationExecutor-0.replicationSource,Indexer_hbaseindexer2-host13.secure.com,16020,1522936844806-host14.secure.com,16020,1523965763579-host14.secure.com,16020,1523966798823]
regionserver.HBaseInterClusterReplicationEndpoint: Can't replicate because of a
local or network error: 
java.io.IOException: Couldn't setup connection for hbase/host13.secure.com@IBMDEV.LOCAL
to hbase/host15.secure.com@IBMDEV.LOCAL  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:665)  at
java.security.AccessController.doPrivileged(Native Method)  at
javax.security.auth.Subject.doAs(Subject.java:422)  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1199)  at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)  at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)  at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:23707)  at
org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:71)  at
org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:179)  at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:825)  at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:444)  Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase  at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:153)  at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:189)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)  at
java.security.AccessController.doPrivileged(Native Method)  at
javax.security.auth.Subject.doAs(Subject.java:422)  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)  at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)  ... 10
more

Here's my Hbase-indexer jaas:

  Client { 
com.sun.security.auth.module.Krb5LoginModule required  
useKeyTab=true  
storeKey=true useTicketCache=false keyTab="/opt/lucidworks-hdpsearch/hbase-indexer/conf/hbasePlusHTTP.service.keytab" principal="hbase/host15.secure.com@IBMDEV.LOCAL"
debug=true;
};

In my zkcli.sh I added the following line :

 ZK_KRB="-Djava.security.auth.login.config=/opt/lucidworks-hdpsearch/hbase-indexer/conf/hbase_indexer.jaas"

And the inserted this option to the PATH variable inside the zkcli.sh from the HDP zookeeper, not the Solr one:

 PATH=$JAVA_HOME/bin:$PATH $JVM $SOLR_ZK_CREDS_AND_ACLS $ZK_KRB
-Dlog4j.configuration=$log4j_config 

In the HBase Indexer logs, I'm seeing anytime I restart HBase this error:

org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase-secure/tokenauth/keys/26

at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)

To overcome it, I'm just adding an "world:anyone:cdrwa" to the new ZNodes, but I'm sure this is not the solution. I'm citing this problem because I'm not sure whether it is related or not to the main problem.

I tried many solutions before asking, but it seems that I'm missing something.

HBase is working. I'm inserting data to Hbase, but changes are not replicated to Solr.

Before kerberization, everything was working perfectly.

Any hint/help/soltion is welcome.

Thanks