SOLR + Kerberos + curl ==> Cannot find key of appropriate type to decrypt AP REP...

[#1]

Hello,

I setup a mini-CDH-5.9 cluster and enabled Kerberos.

Working with HDFS, Hive, Hbase is working fine, the only issue is that I cannot access my SOLR collection via curl command-line. Accessing SOLR collection via Hue is also working.

If I execute e.g.  

curl --negotiate -u : 'http://mgr-node1:8983/solr/mycollection/select?q=*&wt=json&indent=true'

after grabbing a Kerberos ticket , I receive the error:

HTTP Status 403 - GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)

But this enctype is included in the keytab, as well as the supported enctypes in the MIT-KDC:

Keytab:

klist -e -kt /etc/security/cloudera-scm.keytab
Keytab name: FILE:/etc/security/cloudera-scm.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (aes256-cts-hmac-sha1-96)
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (aes128-cts-hmac-sha1-96)
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (des3-cbc-sha1)
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (arcfour-hmac)
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (des-hmac-sha1)
   1 04/26/2017 21:21:22 cloudera-scm@realm.demo (des-cbc-md5)

KDC supported enctypes:

sudo cat /var/kerberos/krb5kdc/kdc.conf | grep supported_enctypes
  supported_enctypes = aes256-cts-hmac-sha1-96 des3-hmac-sha1 aes256-cts arcfour-hmac des-hmac-sha1

Krb client config:

cat /etc/krb5.conf | grep _enctypes
default_tgs_enctypes = aes256-cts-hmac-sha1-96 des3-hmac-sha1 aes256-cts arcfour-hmac des-hmac-sha1
default_tkt_enctypes = aes256-cts-hmac-sha1-96 des3-hmac-sha1 aes256-cts arcfour-hmac des-hmac-sha1
permitted_enctypes = aes256-cts-hmac-sha1-96 des3-hmac-sha1 aes256-cts arcfour-hmac des-hmac-sha1

Any ideas where the error is coming from ?

How to troubleshoot further, so that I can access the collection via curl, e.g. to index new data...

Thanks in advance...

[#2]

After you get the ticket what is the output of klist -ef? That will show the ticket you have and encryption type provided by the KDC that issued the ticket.

[#3]

Hello @mbigelow ,

many thanks for your reply.

Here the output :

## as user cloudera-scm:

-bash-4.2$ klist -ef
Ticket cache: FILE:/tmp/krb5cc_995
Default principal: cloudera-scm@realm.demo

Valid starting       Expires              Service principal
04/28/2017 06:08:22  04/29/2017 06:08:22  krbtgt/realm.demo@realm.demo
        renew until 05/05/2017 06:08:22, Flags: FRI
        Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96

any ideas ?

[#4 · Accepted solution]

at the end it turned out not to be a enctype issue 😉

The problem was that the hostname of the solr server in the curl URL did not match the hostname part of solr's kerberos principal.

After putting the FQDN in the call:

curl --negotiate -u : 'http://mgr-node1.test.demo:8983.........

the error disappeard and everything works nice.