Created 10-15-2017 10:21 AM
We are unable to make queries to collections on Ambari Infra Solr. This same request works on other Ambari 2.5 clusters.
# curl -g --negotiate -u : "http://hostname:8886/solr/ranger_audits/query?debug=query&q=*:*"<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title> </head> <body><h2>HTTP ERROR 403</h2> <p>Problem accessing /solr/ranger_audits/select. Reason: <pre> GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html>
Here is the krb5 debug log showing the duplicate key after setting SOLR_OPTS="$SOLR_OPTS -Dsun.security.krb5.debug=true
.
>>> KrbApReq: authenticate succeed. Krb5Context setting peerSeqNumber to: 907174024 >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType Krb5Context setting mySeqNumber to: 512754846 Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM Entered Krb5Context.acceptSecContext with state=STATE_NEW Looking for keys for: HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM Added key: 17version: 2 Found unsupported keytype (1) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM Found unsupported keytype (3) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM Added key: 18version: 2 >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType Using builtin default etypes for permitted_enctypes default etypes for permitted_enctypes: 18 17 16 23. >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType MemoryCache: add 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM to infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM|HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM MemoryCache: Existing AuthList: #3: 1508065826/137763/263ACEC1894287E10DE785337DF032E1/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM #2: 1508065868/906511/338AF89A3C5C5E73950D89CD559EBEFD/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM #1: 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
But administrative requests work:
# sudo curl -g --negotiate -u : "http://hostname:8886/solr/admin/collections?action=LIST" <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><arr name="collections"><str>fulltext_index</str><str>ranger_audits</str><str>edge_index</str><str>vertex_index</str></arr> </response>
Created 10-16-2017 05:49 AM
Can you try the same curl request with the verbose option and share the response. Looks like the response,
GSSException:Failure unspecified at GSS-API level (Mechanism level:Requestis a replay (34))
implies that the request carries the same token which was previously used for a different request and the connection doesn't seem to be closed. In that case you can destroy the current ticket and try with a fresh ticket.
Created 10-16-2017 08:22 AM
Curl shows the same.
Created 10-16-2017 06:56 PM
@Sean Roberts, did you try curl with -v option ?
Created 11-12-2017 04:48 AM
@Sean Roberts - Was Solr completely initialized when you were hitting it with curl? Did you restart Solr or reload the ranger_audit collection?
One way to get a better error message:
http://hostname:8886/solr/ranger_audits_shard1_replica1/query?debug=query&q=*:*&distrib=false
This will query only a single shard and not try to get bounced around.
On my Ambari Infra server, if I reboot and issue queries before the collections are completely loaded I get the request is a replay hitting /solr/ranger_audits/ but if I hit a single shard I get this error message:
{ "responseHeader":{ "status":503, "QTime":0, "params":{ "q":"*:*", "debug":"query"}}, "error":{ "metadata":[ "error-class","org.apache.solr.common.SolrException", "root-error-class","org.apache.solr.common.SolrException"], "msg":"no servers hosting shard: shard2", "code":503}}
The shard2 could be any shard that is still initializing. You should also be able to see this in the Solr Admin UI -> Cloud and see which shards aren't green.
Created 10-05-2018 12:10 AM
These seems to be bogus replay exception when running solr service.
Changes hadoop-env.sh or solr JVM option with -Dsun.security.krb5.rcache=none should fix the problem.
# # Extra Java runtime options. Empty by default. export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.rcache=none ${HADOOP_OPTS}"