Support Questions
Find answers, ask questions, and share your expertise

Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

Rising Star
I have a Kerberized cluster (HDP 2.5.3) with Solr (5.5.2) that's working well, except for the Banana dashboard.

With a Kerberos ticket, I'm able to access the Solr UI and run queries. Without a Kerberos ticket, the Solr UI returns `Authentication required` as expected. However, when I have a ticket and try to access the Banana dashboard the following error is returned:

11704-solr-banana-403.png

Steve Loughran's excellent book, Kerberos on Hadoop, identifies the possible causes of a `request is a replay` error:

The KDC is seeing too many attempts by the caller to authenticate as a specific principal, assumes some kind of attack and rejects the request. This can happen if you have too many processes/ nodes all sharing the same principal. Fix: make sure you have service/_HOST@REALM principals for all the services, rather than simple service@REALM principals.

The timestamps of the systems are out of sync, so it looks like an old token be re-issued. Check them all, including that of the KDC, make sure NTP is working, etc, etc.

All the Solr principals are host-specific and NTP is running everywhere.

Has anyone seen seen this before? Or have suggestions why Banana and Kerberos aren't working well together?

6 REPLIES 6

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

Guru

Hello @Alex Woolford,

From my Kerberos experience, fixing this kind of error can sometimes be annoying. The quickest way to know what is going on would be to capture the network packet trace (tcpdump) between browser and Solr/Banana Service. Once captured, look for Kerberos packets to understand who is sending what & how many times. Hope this helps !

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

New Contributor

I recently ran into the same issue and believe the error is caused by the kerberos using the same transaction to authenticate first to solr then to banana.

I enabled solr to provide verbose kerberos logging by adding the following to the advanced solr-env under amabri.

SOLR_OPTS="$SOLR_OPTS -Xss256k -Dsun.security.krb5.debug=true

Now when i attempt to login to banana using my kerberos credentials @ http://pr1.bmeu.com:8983/solr/banana/

I get the following in the solr-8983.console.log

Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Found unsupported keytype (3) for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Added key: 18version: 0
Added key: 17version: 0
Added key: 23version: 0
Added key: 16version: 0
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
MemoryCache: add 1490012724/056454/5F6E3D8C7220D1427AC2182369D70085/peter@TCAD.BMEU.COM to peter@TCAD.BMEU.COM|HTTP/pr1.bmeu.com@TCAD.BMEU.COM
MemoryCache: Existing AuthList:
#2: 1490012709/098313/235E39D12B9D614AA472FEE77755F391/peter@TCAD.BMEU.COM
#1: 1490012716/216256/1D970AC4EAC7783169D2FB7256251703/peter@TCAD.BMEU.COM


>>> KrbApReq: authenticate succeed.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
>>>Delegated Creds have pname=peter@TCAD.BMEU.COM sname=krbtgt/TCAD.BMEU.COM@TCAD.BMEU.COM authtime=20170320110112Z starttime=20170320122523Z endtime=20170320210112ZrenewTill=20170327110109Z
Krb5Context setting peerSeqNumber to: 755538422
Krb5Context setting mySeqNumber to: 755538422
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Found unsupported keytype (3) for HTTP/pr1.bmeu.com@TCAD.BMEU.COM
Added key: 18version: 0
Added key: 17version: 0
Added key: 23version: 0
Added key: 16version: 0
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.ArcFourHmacEType
MemoryCache: add 1490012724/056454/5F6E3D8C7220D1427AC2182369D70085/peter@TCAD.BMEU.COM to peter@TCAD.BMEU.COM|HTTP/pr1.bmeu.com@TCAD.BMEU.COM
MemoryCache: Existing AuthList:
#3: 1490012709/098313/235E39D12B9D614AA472FEE77755F391/peter@TCAD.BMEU.COM
#2: 1490012716/216256/1D970AC4EAC7783169D2FB7256251703/peter@TCAD.BMEU.COM
#1: 1490012724/056454/5F6E3D8C7220D1427AC2182369D70085/peter@TCAD.BMEU.COM


From the log it seems that the attempt to add add 1490012724/056454/5F6E3D8C7220D1427AC2182369D70085/peter@TCAD.BMEU.COM to the kdc cache fails as the transaction is already saved there..

Thus we get the request is a replay error

Workaround ..

Create a new file called banana-jetty-context.xml under /opt/lucidworks-hdpsearch/solr/server/contexts

and populate it with the following

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure>
  <Set name="contextPath"><Property name="hostContext" default="/banana"/></Set>
  <Set name="war"><Property name="jetty.base"/>/solr-webapp/webapp/banana</Set>
  <Set name="defaultsDescriptor"><Property name="jetty.base"/>/etc/webdefault.xml</Set>
  <Set name="extractWAR">false</Set>
</Configure>

Restart solr from ambari

now you should be able to log into banana bypassing solr

http://pr1.bmeu.com:8983/banana/

13797-screenshot-20170320-123732.png

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

New Contributor

Hi,

I have the same problem on Banana after having kerberized SOLR (5.5.2).

13873-capture-psg-banana1.png

Your workaround doesn't work on my SOLR plateform. I created a new file called "banana-jetty-context.xml" under "/opt/lucidworks-hdpsearch/solr/server/contexts" and restarting Ambari server.

Now, when i try to access the Banana dashboard the following error is returned :

13840-capture-psg-banana.png

Do you have an idea for this issue ?

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

New Contributor

hi psg91

did you populate the new file as mentioned above ?

also check the file permissions on the new file and restart solr

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

@Psg91 Psg91

Banana context is missing the following line, which solr context has:

<Configure class="org.eclipse.jetty.webapp.WebAppContext">

Re: Banana returns a `request is a replay` error on a Kerberized cluster and yet the Solr UI works fine

Cloudera Employee

@pryan and @James Switzer are spot on but in HDP 2.6.0.3/Ambari 2.5.0.3 environment - we needed default="/solr/banana" - see the revised banana-jetty-context.xml under /opt/lucidworks-hdpsearch/solr/server/contexts below

<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath"><Property name="hostContext" default="/solr/banana"/></Set>
  <Set name="war"><Property name="jetty.base"/>/solr-webapp/webapp/banana</Set>
  <Set name="defaultsDescriptor"><Property name="jetty.base"/>/etc/webdefault.xml</Set>
  <Set name="extractWAR">false</Set>
</Configure>