Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn ATS replay request on security enabled cluster.

Solved Go to solution
Highlighted

Yarn ATS replay request on security enabled cluster.

Expert Contributor

Hello community,

I'have a cluster with kerberos and after a restart I'm having the following error when trying to reach the ATS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /applicationhistory. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr /><i><small>Powered by Jetty://ll></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>


</body>
</html>
I've tried with diferent principals but the issue persist.

The resource manager and the rest of the spnego authenticated web consoles still working properly.

Any idea about what is going on?

Thank you in advance

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Yarn ATS replay request on security enabled cluster.

Expert Contributor

Hello @Vipin Rathor,

An apology for the delay in the answer, finally I solved it, as you said the problem with the replay was that he was trying to authenticate multiple times in a very short time, this was caused by curl and the -L parameter, for some reason curl wasn't storing the session cookie, I fixed it using -c <file path> -b <file path> parameter to store the cookie.

Thank you.

2 REPLIES 2

Re: Yarn ATS replay request on security enabled cluster.

Guru

Hello @Juan Manuel Nieto,

The 'request is a replay' usually means that your Kerberos KDC is processing two same requests (almost) at the same time and 'thinks' that the second request is actually a replay attack than a legitimate request. Therefore it will drop the second request.

Things to check here:

1. KDC logs - try to find out for which requests and what principal, it is complaining about

2. Application log - Check YARN ATS log for any possible error w.r.t. Kerberos

3. Are you using same keytab with multiple services? (apart from spnego.service.keytab which is shared across services on a node). If yes, please try and use different keytabs for different services.

For the logs, it'll be worth to increase the Kerberos debug level by setting up "-Dsun.security.krb5.debug=true" in the JVM parameters and also try 'export KRB5_TRACE=/tmp/somefile' on the client-side before running a curl request (using browser won't help much).

Hope this helps !

Re: Yarn ATS replay request on security enabled cluster.

Expert Contributor

Hello @Vipin Rathor,

An apology for the delay in the answer, finally I solved it, as you said the problem with the replay was that he was trying to authenticate multiple times in a very short time, this was caused by curl and the -L parameter, for some reason curl wasn't storing the session cookie, I fixed it using -c <file path> -b <file path> parameter to store the cookie.

Thank you.