question Yarn ATS replay request on security enabled cluster. in Archives of Support Questions (Read Only)

Yarn ATS replay request on security enabled cluster.

juan_manuel_nie — Wed, 19 Apr 2017 00:57:04 GMT

Hello community,

I'have a cluster with kerberos and after a restart I'm having the following error when trying to reach the ATS

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /applicationhistory. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))</pre></p><hr /><i><small>Powered by Jetty://ll></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>


</body>
</html>

I've tried with diferent principals but the issue persist.

The resource manager and the rest of the spnego authenticated web consoles still working properly.

Any idea about what is going on?

Thank you in advance

Re: Yarn ATS replay request on security enabled cluster.

VR46 — Mon, 24 Apr 2017 20:21:15 GMT

Hello @Juan Manuel Nieto,

The 'request is a replay' usually means that your Kerberos KDC is processing two same requests (almost) at the same time and 'thinks' that the second request is actually a replay attack than a legitimate request. Therefore it will drop the second request.

Things to check here:

1. KDC logs - try to find out for which requests and what principal, it is complaining about

2. Application log - Check YARN ATS log for any possible error w.r.t. Kerberos

3. Are you using same keytab with multiple services? (apart from spnego.service.keytab which is shared across services on a node). If yes, please try and use different keytabs for different services.

For the logs, it'll be worth to increase the Kerberos debug level by setting up "-Dsun.security.krb5.debug=true" in the JVM parameters and also try 'export KRB5_TRACE=/tmp/somefile' on the client-side before running a curl request (using browser won't help much).

Hope this helps !

Re: Yarn ATS replay request on security enabled cluster.

juan_manuel_nie — Wed, 17 May 2017 22:15:59 GMT

Hello @Vipin Rathor,

An apology for the delay in the answer, finally I solved it, as you said the problem with the replay was that he was trying to authenticate multiple times in a very short time, this was caused by curl and the -L parameter, for some reason curl wasn't storing the session cookie, I fixed it using -c <file path> -b <file path> parameter to store the cookie.

Thank you.