question Re: Solr "Request is a replay" (Ambari Infra Solr 2.5) in Support Questions

Solr "Request is a replay" (Ambari Infra Solr 2.5)

sroberts — Sun, 15 Oct 2017 17:21:25 GMT

We are unable to make queries to collections on Ambari Infra Solr. This same request works on other Ambari 2.5 clusters.

# curl -g --negotiate -u : "http://hostname:8886/solr/ranger_audits/query?debug=query&q=*:*"<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /solr/ranger_audits/select. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
</body>
</html>

Here is the krb5 debug log showing the duplicate key after setting SOLR_OPTS="$SOLR_OPTS -Dsun.security.krb5.debug=true.

>>> KrbApReq: authenticate succeed.
Krb5Context setting peerSeqNumber to: 907174024
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 512754846
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Added key: 17version: 2
Found unsupported keytype (1) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Found unsupported keytype (3) for HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
MemoryCache: add 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM to infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM|HTTP/hostname.domain.com@MYCLUSTER.DOMAIN.COM
MemoryCache: Existing AuthList:
#3: 1508065826/137763/263ACEC1894287E10DE785337DF032E1/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
#2: 1508065868/906511/338AF89A3C5C5E73950D89CD559EBEFD/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM
#1: 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/hostname.domain.com@MYCLUSTER.DOMAIN.COM

But administrative requests work:

# sudo curl -g --negotiate -u : "http://hostname:8886/solr/admin/collections?action=LIST"
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><arr name="collections"><str>fulltext_index</str><str>ranger_audits</str><str>edge_index</str><str>vertex_index</str></arr>
</response>

Re: Solr "Request is a replay" (Ambari Infra Solr 2.5)

vsuvagia — Mon, 16 Oct 2017 12:49:46 GMT

@Sean Roberts,

Can you try the same curl request with the verbose option and share the response. Looks like the response,

GSSException:Failure unspecified at GSS-API level (Mechanism level:Requestis a replay (34))

implies that the request carries the same token which was previously used for a different request and the connection doesn't seem to be closed. In that case you can destroy the current ticket and try with a fresh ticket.

Re: Solr "Request is a replay" (Ambari Infra Solr 2.5)

sroberts — Mon, 16 Oct 2017 15:22:28 GMT

Curl shows the same.

Re: Solr "Request is a replay" (Ambari Infra Solr 2.5)

vsuvagia — Tue, 17 Oct 2017 01:56:21 GMT

@Sean Roberts, did you try curl with -v option ?

Re: Solr "Request is a replay" (Ambari Infra Solr 2.5)

risdenk — Sun, 12 Nov 2017 12:48:15 GMT

@Sean Roberts - Was Solr completely initialized when you were hitting it with curl? Did you restart Solr or reload the ranger_audit collection?

One way to get a better error message:

http://hostname:8886/solr/ranger_audits_shard1_replica1/query?debug=query&q=*:*&distrib=false

This will query only a single shard and not try to get bounced around.

On my Ambari Infra server, if I reboot and issue queries before the collections are completely loaded I get the request is a replay hitting /solr/ranger_audits/ but if I hit a single shard I get this error message:

{
  "responseHeader":{
    "status":503,
    "QTime":0,
    "params":{
      "q":"*:*",
      "debug":"query"}},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"no servers hosting shard: shard2",
    "code":503}}

The shard2 could be any shard that is still initializing. You should also be able to see this in the Solr Admin UI -> Cloud and see which shards aren't green.

Re: Solr "Request is a replay" (Ambari Infra Solr 2.5)

xyao — Fri, 05 Oct 2018 07:10:59 GMT

These seems to be bogus replay exception when running solr service.

Changes hadoop-env.sh or solr JVM option with -Dsun.security.krb5.rcache=none should fix the problem.

# # Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.rcache=none ${HADOOP_OPTS}"