Kerberized Solr Loadbalancer

[#1]

Hi everybody,

I'm trying to configure a loadbalancer for 3 solr Instances Cloudera CDH 5.10.2 .

my deployment is:

node1.mydomain

node2.mydomain

node3.mydomain

i'would like to configure a loadbalancer on an external machine with a FQDN like solr.mydomain.

I've configured the loadbalancer on clouderamanager and i've generated the missing principal for the load balancer and configured these environment variabiles:

SOLR_AUTHENTICATION_TYPE=kerberos
SOLR_AUTHENTICATION_KERBEROS_PRINCIPAL=*

Now i have this principals:

solr/node1.mydomain@MYREALM

solr/node2.mydomain@MYREALM

solr/node3.mydomain@MYREALM

HTTP/node1.mydomain@MYREALM

HTTP/node2.mydomain@MYREALM

HTTP/node3.mydomain@MYREALM

HTTP/solr.mydomain@MYREALM

Everything seems to be configured to work but if i try to connect i got authentication error.

I've tryed with curl and i got this response:

# curl -i --negotiate -u : https://solrs.mydomain:8985/solr --location -k

403 Frobidden

GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)

If i connect directly to a single instance it works:

curl -i --negotiate -u : https://node1.mydomain:8985/solr --location -k

I've also verified the principals in the solr.keytab of every node with ktutils:

slot KVNO Principal
---- ---- ---------------------------------------------------------------------
1 1 HTTP/node1.mydomain@MYREALM
2 1 HTTP/solr.mydomain@MYREALM
3 1 solr/node1.mydomain@MYREALM

Did someone have the same issue, i'm missing something ?

Thank you so much in advance for your help.

Ivan

[#2]

I have the same configuration with HAProxy and it works fine.

To be honest, I do not specify any environment variable, just running kinit and the issuing the curl command.

It is probably something with your loadbalancer configuration.

How it is configured.

In the mean time, can you check keytab with "klist -ket solr.keytab" to verify that you have the same encryption type?

[#3]

Hi GeKas
Same Encryption type arcfour-hmac (ActiveDirectory):

klist -ket solr.keytab
Keytab name: FILE:solr.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 03/01/2018 16:27:50 HTTP/node1.mydomain@MYREALM (arcfour-hmac)
1 03/01/2018 16:27:50 HTTP/solrs.mydomain@MYREALM (arcfour-hmac)
1 03/01/2018 16:27:50 solr/node1.mydomain@MYREALM (arcfour-hmac)

(to be clear the node names and the realm are fake, they are just a placeholder to mask my real hosts 🙂 )
(The environment variable is for solr services)

i'm using haproxy too.

Do you have a "template" for haproxy configuration maybe i'm missing something?

[#4]

I believe it is obvious that names are fake 🙂 🙂 🙂

I am not sure this question is appropriate for this forum, but anyway.

The following configuration works. SSL from client terminates on HAProxy, while HAProxy to solr server opens a new SSL connection.

bind :8985 ssl crt {your certificate, key and any CA here}
mode http
option httpclose
option http-server-close
option forwardfor
server solrserver node1.mydomain:8985 check ssl verify required ca-file {Your CA certificates here}

If you still have issues, then the problem is not on HAProxy and you should check your keytabs (typo error, etc).

You can also verigy that keytab contents are ok, by using the commands below:

kinit -kt solr.keytab HTTP/node1.mydomain@MYREALM
kinit -kt solr.keytab HTTP/solrs.mydomain@MYREALM
kinit -kt solr.keytab solr/node1.mydomain@MYREALM

If any one fails, then you should recreate the keytab.

When creating the keytab with ktutil, this command does not guarantee that the contents of the keytab (encrypted password) are correct. It does not connect to KDC to verify it. It is just encrypting the password you provide (or read from another keytab file). In order to verify it, you should test it with kinit.

Hope that helps.

[#5]

Ty again GeKas,

i've verified haproxy configuration and its like youre example.

kinit with all principals in the keytab works without errors.

Again doing a curl on the balancer i've:

GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)

on server side i've this exception:

2018-03-02 19:16:05,261 DEBUG org.apache.hadoop.security.authentication.server.AuthenticationFilter: Request [https://solr.mydomain:8985/solr/] triggering authentication
2018-03-02 19:16:05,261 DEBUG org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:398)
at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:538)
at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:413)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:612)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:365)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347)
... 15 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:102)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281)
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
... 22 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:408)
at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:91)
at sun.security.krb5.internal.crypto.ArcFourHmacEType.decrypt(ArcFourHmacEType.java:100)
... 28 more

[#6 · Accepted solution]

With the error you provided, I woud definitely look again the keytab entries, one by one.

This keytab should reside in SOLR's execution directory. Assuming you are using Cloudera, under "/var/run/cloudera-scm-agent/process/{id}-SOLR...

This error, 90% of the times mean that keytab is not correct.

[#7]

Solved,

The DNS entry was wrong ... solrs.mydomain was a CNAME and not a A Record.

Thanks again for your help!