Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SolrCloud does not work in CDH 5.3.2 + Kerberos

Highlighted

SolrCloud does not work in CDH 5.3.2 + Kerberos

Explorer

After upgrade from 5.3.1 to 5.3.2 it is not possible to communcate with SolrCloud using solrj java library.

 

On client side I getting exception

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://catnn002:8983/solr/prod_cases returned non ok status:403, message:Forbidden

 Seems, that something was changed in Solr Tomcat confiration or code. It does not transform host names into canonical names. But at the same time zookeeper keeps hostnames in short form (/solr/live_nodes)

 

In server logs the exception is

2015-03-23 05:50:19,885 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Failure unspecified
at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument
(400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:520)
        at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:277)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP -
 RC4 with HMAC)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:899)
        at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:550)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:366)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:348)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:348)
        ... 18 more
Caused by: KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:288)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:159)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
        ... 29 more

 

Before upgrade I had running collection with 2 shards and 2 Solr instances. After upgrade solr server logs is full with following messages

2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "GET /solr/admin/cores?action=PREPRECOVERY&core=prod_cases_shard2_replica1&nodeName=catnn001%3A8983_solr&coreNodeName=core_node1&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true&wt=javabin&version=2 HTTP/1.1[\r][\n]"
2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrServer] 1.0[\r][\n]"
2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "Host: catnn002:8983[\r][\n]"
2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "Connection: Keep-Alive[\r][\n]"
2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "Authorization: Negotiate .....[\r][\n]"
2015-03-22 11:27:33,116 DEBUG org.apache.http.wire: >> "[\r][\n]"

2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "HTTP/1.1 403 Forbidden[\r][\n]"
2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "Server: Apache-Coyote/1.1[\r][\n]"
2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly[\r][\n]"
2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "Content-Type: text/html;charset=utf-8[\r][\n]"
2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "Content-Length: 1289[\r][\n]"
2015-03-22 11:27:33,125 DEBUG org.apache.http.wire: << "Date: Sun, 22 Mar 2015 10:27:33 GMT[\r][\n]"

....
HTTP Status 403 - GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - RC4 with HMAC)
.....

 

1 REPLY 1

Re: SolrCloud does not work in CDH 5.3.2 + Kerberos

New Contributor

looking at the error,ther is issue with your hostname and FQDN. Try to set "HOSTNAME=hostname.example.com" on the "Solr Service Environment Advanced Configuration Snippet (Safety Valve)" For each instance. This may fix the issue.

Don't have an account?
Coming from Hortonworks? Activate your account here