Cloudera Labs
Provide feedback on Cloudera Labs
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cdh5.14.2 + Phoenix QueryServer 4.14.1 + Kerberos

cdh5.14.2 + Phoenix QueryServer 4.14.1 + Kerberos

New Contributor

I'm trying to setup Apache Phoenix QueryServer in secure HBase environment.

My hbase-site.xml is:

<configuration>
  <property>
    <name>hbase.regionserver.wal.codec</name>
    <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>zk1,zk2,zk3</value>
  </property>
<property>
    <name>hbase.security.authentication</name>
    <value>kerberos</value>
</property>
<property>
    <name>phoenix.queryserver.keytab.file</name>
    <value>/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab</value>
</property>
<property>
    <name>phoenix.queryserver.kerberos.principal</name>
    <value>rwqueryserver/_HOST@FOO.BAR</value>
</property>
<property>
    <name>phoenix.queryserver.http.keytab.file</name>
    <value>/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab</value>
</property>
<property>
    <name>phoenix.queryserver.http.kerberos.principal</name>
    <value>rwqueryserver/_HOST@FOO.BAR</value>
</property>
</configuration>

Query server is starting without any problems.

2018-12-12 09:13:07,353 INFO org.apache.phoenix.queryserver.server.QueryServer: Login successful.

I checked KDC side and I can see logins from the principal used for server/client connections. No errors on KDC side as well. Thin client command is:

./sqlline-thin.py 'http://dns-of-query-server:8765;principal="rwqueryserver/dns-of-query-server@DATASYS.CF.WTF";keytab="/usr/lib/apache-phoenix-queryserver/bin/phoenix.keytab"'

I tried to use user's principal and server's principal - situation is the same.

java.lang.RuntimeException: Failed to execute HTTP Request, got HTTP/404

From the queryserver log:

2018-12-12 09:15:30,987 WARN org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService: 
GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:906)
        at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:556)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.SpnegoLoginService.login(SpnegoLoginService.java:137)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.LoginAuthenticator.login(LoginAuthenticator.java:61)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.authentication.SpnegoAuthenticator.validateRequest(SpnegoAuthenticator.java:99)
        at org.apache.phoenix.shaded.org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.apache.phoenix.shaded.org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.apache.phoenix.shaded.org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.apache.phoenix.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
        ... 19 more

When I'm trying to use non-existing principal I'm getting different errors. I checked JCE - it's installed.

jrunscript -e 'print (javax.crypto.Cipher.getMaxAllowedKeyLength("AES") >= 256);'
true

Can you advice anything ?

3 REPLIES 3
Highlighted

Re: cdh5.14.2 + Phoenix QueryServer 4.14.1 + Kerberos

Master Collaborator
Can you post your /etc/krb5.conf? And your setting of KDC what kind of ciphers are supported?

Re: cdh5.14.2 + Phoenix QueryServer 4.14.1 + Kerberos

New Contributor
[libdefaults]
dns_lookup_kdc = false
dns_uri_lookup = false
ticket_lifetime = 24h
renew_lifetime = 7d
default_tgs_enctypes = "aes256-cts-hmac-sha1-96 -des -des3 -rc4 -camellia"
default_tkt_enctypes = "aes256-cts-hmac-sha1-96 -des -des3 -rc4 -camellia"

default_ccache_name = KEYRING:persistent:%{uid}
default_realm = FOO.BAR

[logging]
default = FILE:/var/log/krb5libs.log

[realms]
FOO.BAR = {
    kdc = krb-kdc001-server
    admin_server = krb-kdc001-server
}

From krb serever

[realms]
FOO.BAR = {
    max_life = 12h 0m 0s
    max_renewable_life = 7d 0h 0m 0s
    master_key_type = aes256-cts
    supported_enctypes = aes256-cts:normal
    dict_file = /usr/share/dict/words
}

Re: cdh5.14.2 + Phoenix QueryServer 4.14.1 + Kerberos

New Contributor

During queryserver startup I ca see

19/02/05 12:12:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/02/05 12:12:52 INFO server.QueryServer: Login successful.
19/02/05 12:12:52 INFO metrics.MetricsSystemLoader: No metrics implementation available on classpath. Using No-op implementation

This means that queryserver itself is able to use supplied keytab file and authinticate with Kerberos.

When I'm trying to connect with client, using keytab I'm still getting mentioned error:

GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

but client tries to use keytab created in the same way as server's keytab.

 

Also I canged java to OpenJDK