Support Questions

juan_manuel_nie · ‎04-18-2017

Hello community,

I'have a cluster with kerberos and after a restart I'm having the following error when trying to reach the ATS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /applicationhistory. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr /><i><small>Powered by Jetty://ll></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>


</body>
</html>

I've tried with diferent principals but the issue persist.

The resource manager and the rest of the spnego authenticated web consoles still working properly.

Any idea about what is going on?

Thank you in advance

juan_manuel_nie · ‎05-17-2017

Hello @Vipin Rathor,

An apology for the delay in the answer, finally I solved it, as you said the problem with the replay was that he was trying to authenticate multiple times in a very short time, this was caused by curl and the -L parameter, for some reason curl wasn't storing the session cookie, I fixed it using -c <file path> -b <file path> parameter to store the cookie.

Thank you.

View solution in original post

VR46 · ‎04-24-2017

Hello @Juan Manuel Nieto,

The 'request is a replay' usually means that your Kerberos KDC is processing two same requests (almost) at the same time and 'thinks' that the second request is actually a replay attack than a legitimate request. Therefore it will drop the second request.

Things to check here:

1. KDC logs - try to find out for which requests and what principal, it is complaining about

2. Application log - Check YARN ATS log for any possible error w.r.t. Kerberos

3. Are you using same keytab with multiple services? (apart from spnego.service.keytab which is shared across services on a node). If yes, please try and use different keytabs for different services.

For the logs, it'll be worth to increase the Kerberos debug level by setting up "-Dsun.security.krb5.debug=true" in the JVM parameters and also try 'export KRB5_TRACE=/tmp/somefile' on the client-side before running a curl request (using browser won't help much).

Hope this helps !

juan_manuel_nie · ‎05-17-2017

Hello @Vipin Rathor,

An apology for the delay in the answer, finally I solved it, as you said the problem with the replay was that he was trying to authenticate multiple times in a very short time, this was caused by curl and the -L parameter, for some reason curl wasn't storing the session cookie, I fixed it using -c <file path> -b <file path> parameter to store the cookie.

Thank you.

Cloudera Community

Support Questions

Yarn ATS replay request on security enabled cluster.