Support Questions
Find answers, ask questions, and share your expertise

Solr "Request is a replay" (Ambari Infra Solr 2.5)

We are unable to make queries to collections on Ambari Infra Solr. This same request works on other Ambari 2.5 clusters.

# curl -g --negotiate -u : "http://hostname:8886/solr/ranger_audits/query?debug=query&q=*:*"<html>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /solr/ranger_audits/select. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

Here is the krb5 debug log showing the duplicate key after setting SOLR_OPTS="$SOLR_OPTS

>>> KrbApReq: authenticate succeed.
Krb5Context setting peerSeqNumber to: 907174024
>>> EType:
Krb5Context setting mySeqNumber to: 512754846
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/
Added key: 17version: 2
Found unsupported keytype (1) for HTTP/
Found unsupported keytype (3) for HTTP/
Added key: 18version: 2
>>> EType:
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType:
MemoryCache: add 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/ to infra-solr/|HTTP/
MemoryCache: Existing AuthList:
#3: 1508065826/137763/263ACEC1894287E10DE785337DF032E1/infra-solr/
#2: 1508065868/906511/338AF89A3C5C5E73950D89CD559EBEFD/infra-solr/
#1: 1508065886/914380/B8B23803754E028D7923075AFEB12AAC/infra-solr/

But administrative requests work:

# sudo curl -g --negotiate -u : "http://hostname:8886/solr/admin/collections?action=LIST"
<?xml version="1.0" encoding="UTF-8"?>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><arr name="collections"><str>fulltext_index</str><str>ranger_audits</str><str>edge_index</str><str>vertex_index</str></arr>

Cloudera Employee

@Sean Roberts,

Can you try the same curl request with the verbose option and share the response. Looks like the response,

GSSException:Failure unspecified at GSS-API level (Mechanism level:Requestis a replay (34))

implies that the request carries the same token which was previously used for a different request and the connection doesn't seem to be closed. In that case you can destroy the current ticket and try with a fresh ticket.

Curl shows the same.

Cloudera Employee

@Sean Roberts, did you try curl with -v option ?


@Sean Roberts - Was Solr completely initialized when you were hitting it with curl? Did you restart Solr or reload the ranger_audit collection?

One way to get a better error message:


This will query only a single shard and not try to get bounced around.

On my Ambari Infra server, if I reboot and issue queries before the collections are completely loaded I get the request is a replay hitting /solr/ranger_audits/ but if I hit a single shard I get this error message:

    "msg":"no servers hosting shard: shard2",

The shard2 could be any shard that is still initializing. You should also be able to see this in the Solr Admin UI -> Cloud and see which shards aren't green.

Rising Star

These seems to be bogus replay exception when running solr service.

Changes or solr JVM option with should fix the problem.

# # Extra Java runtime options.  Empty by default.