Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Use solr with kerberos

Use solr with kerberos

Rising Star

Hi @Jonas Straub and @Ali Bajwa,

I read this article, but if I run any curl query solr returned:

WARN  org.apache.hadoop.security.authentication.server.AuthenticationFilter  [   ] – Authentication exception: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)

I have set correctly jaas.conf file and I have upload on zookeeper this:

/zkcli.sh -zkhost tst-master1:2181,tst-master3:2181,tst-master2:2181 -cmd put /solr/security.json '{"authentication":{"class": "org.apache.solr.security.KerberosPlugin"}}'

I have omitted the part related to ranger because it is not used in this cluster.

thanks in advance.

P.S.: This is my klist output:

[solr@tst-master1 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_6002
Default principal: solr/tst-master1@MYREALM.COM
Valid starting     Expires            Service principal
06/08/16 13:27:53  06/09/16 13:27:53  krbtgt/MYREALM.COM@MYREALM.COM
06/08/16 13:28:03  06/09/16 13:27:53  HTTP/tst-master1@MYREALM.COM
7 REPLIES 7

Re: Use solr with kerberos

Rising Star
[solr@tst-master1 ~]$ cat /opt//lucidworks-hdpsearch/solr/bin/jaas.conf
Client {
     com.sun.security.auth.module.Krb5LoginModule required
     useKeyTab=true
     keyTab="/etc/security/keytabs/solr.service.keytab"
     storeKey=true
     debug=true
     doNotPrompt=true
     principal="solr/tst-master1@MYREAL.COM";
};



Re: Use solr with kerberos

Contributor
  1. Make sure the JVM that Solr is using has the JCE Unlimited Strength Keys Policy installed
  2. Check the Solr or ZK logs and send any relevant output as there might be more detail there
  3. Try adding this JVM parameter to the Solr/ZK JVMs to increase debug output:
-Dsun.security.krb5.debug=true

Re: Use solr with kerberos

Rising Star

Hi @Eric Walk,

I have enabled debug mode for kerberos and I see this in the solr log:

Found ticket for solr/tst-master1@MYREAL.COM to go to krbtgt/MYREALM.COM@MYREALM.COM expiring on Fri Jun 10 09:00:52 UTC 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 16 23 1 3..
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=tst-kerb2. UDP:88, timeout=30000, number of retries =3, #bytes=724
>>> KDCCommunication: kdc=tst-kerb2. UDP:88, timeout=30000,Attempt =1, #bytes=724
>>> KrbKdcReq send: #bytes read=729
>>> KdcAccessibility: remove tst-kerb2.:88
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 971277641

Then solr use the correct ticket for comunicate with kerberos, but when I lanch a curl api request it return me this:

solr@tst-master1 ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_6002
Default principal: HTTP/tst-master1@MYREALM.COM
Valid starting     Expires            Service principal
06/09/16 11:02:39  06/10/16 11:02:39  krbtgt/MYREALM.COM@MYREALM.COM
06/09/16 11:03:12  06/10/16 11:02:39  HTTP/tst-master1@MYREALM.COM
[solr@tst-master1 ~]$ curl --negotiate -u : "http://tst-master1:8983/solr/testCollection/select?q=*%3A*&row=0&wt=json"
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /solr/testCollection/select. Reason:
<pre>    GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>

Any ideas?

Re: Use solr with kerberos

Contributor

@Davide Isoardi,

Can you please send the complete log from solr at the time if the error, including krb5 debug?

That error message is generic and the logs you sent don't show it happening. The message means that either curl isn't sending a ticket, or solr can't interpret it or verify it.

Re: Use solr with kerberos

Rising Star

When I lanch a curl api request (with kerberos debag mode on) in solr log I see this:

12793190 [qtp139012968-19] WARN  org.apache.hadoop.security.authentication.server.AuthenticationFilter  [   ] – Authentication exception: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)
	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:507)
	at org.apache.solr.security.KerberosFilter.doFilter(KerberosFilter.java:50)
	at org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:120)
	at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:261)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:203)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:497)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:744)
Caused by: GSSException: No valid credentials provided (Mechanism level: Attempt to obtain new ACCEPT credentials failed!)
	at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:81)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:126)
	at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:192)
	at sun.security.jgss.spnego.SpNegoMechFactory.getCredentialElement(SpNegoMechFactory.java:141)
	at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:192)
	at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:406)
	at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:75)
	at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:159)
	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:356)
	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
	... 26 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain Princpal Name for authentication 
	at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:800)
	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:671)
	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784)
	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
	at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721)
	at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718)
	at javax.security.auth.login.LoginContext.login(LoginContext.java:590)
	at sun.security.jgss.GSSUtil.login(GSSUtil.java:255)
	at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:334)
	at sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:76)
	at sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:73)
	... 38 more

Re: Use solr with kerberos

Rising Star

No report by kerberos; sometimes in solr log I see this:

Entered Krb5Context.initSecContext with state=STATE_IN_PROCESS
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting peerSeqNumber to: 1040665739
Krb5Context.unwrap: token=[05 04 01 ff 00 0c 00 00 00 00 00 00 3e 07 4c 8b 01 01 00 00 bd af ae ba 1a 96 94 c6 0b a3 b6 e5 ]
Krb5Context.unwrap: data=[01 01 00 00 ]
Krb5Context.wrap: data=[01 01 00 00 ]
Krb5Context.wrap: token=[05 04 00 ff 00 0c 00 00 00 00 00 00 1a 79 57 37 01 01 00 00 ea 2b 94 71 78 d2 0a 34 fa 59 cc 97 ]
Found ticket for solr/tst-master1@MYREALM.COM to go to krbtgt/MYREALM.COM@MYREALM.COM expiring on Fri Jun 10 09:00:52 UTC 2016
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23 1 3.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
>>> KrbKdcReq send: kdc=tst-kerb1. UDP:88, timeout=30000, number of retries =3, #bytes=724
>>> KDCCommunication: kdc=tst-kerb1. UDP:88, timeout=30000,Attempt =1, #bytes=724
>>> KrbKdcReq send: #bytes read=729
>>> KdcAccessibility: remove tst-kerb1.:88
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 609489532
Created InitSecContextToken:
0000: 01 00 6E 82 02 86 30 82   02 82 A0 03 02 01 05 A1  ..n...0.........
... more
Highlighted

Re: Use solr with kerberos

Expert Contributor

@Davide Isoardi:

try adding this:

-Djava.security.krb5.conf=/etc/krb5.conf

to your solr process. See if you get something else rather than

Service ticket not found in the subject