Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

benhadoop — Fri, 07 Oct 2016 19:32:08 GMT

Hi Community,

I am experiencing weird errors with Oozie, YARN and the Application timeline server.

Running the Ambari Service checks for Oozie, Oozie is not able to get Delegation tokens for the ATS via the resourcemanager due to this error:

in resourcemanager log:

2016-10-07 13:26:43,460 WARN  security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(908)) - Unable to add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.40.11.42:8190, Ident: (owner=ambari-qa, renewer=yarn, realUser=oozie, issueDate=1475839603327, maxDate=1476444403327, sequenceNumber=122, masterKeyId=102)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:475)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:78)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:904)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:881)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: HTTPS hostname wrong:  should be <10.40.11.42>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:188)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:216)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:405)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:387)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientRetryOpForOperateDelegationToken.run(TimelineClientImpl.java:699)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:185)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:462)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:409)
	at org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81)
	at org.apache.hadoop.security.token.Token.renew(Token.java:385)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:597)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:594)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:592)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:461)
	... 6 more

I believe the interesting part is:

Caused by: java.io.IOException: HTTPS hostname wrong:  should be <10.40.11.42>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)

The config is:

- HDP 2.5.0.0 and Ambari 2.4.0.1

- HTTPs is activated for both Hadoop (HDFS, YARN, ATS) and Oozie. The certificates include the hostname(s) of the server

- Kerberos is activated

- The cluster is multihomed, but this communication only happens internally

- hadoop.security.token.service.use_ip is already set to false

Do you have any idea, which config I could adjust to fix this? Or is it a bug?

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

falbani — Fri, 07 Oct 2016 19:42:48 GMT

@Roland Simonis Oozie will fail with this TIMELINE_DELEGATION_TOKEN when hadoop.security.token.service.use_ip=false. This is only supported in HDP 2.5 of Oozie and not in prior versions. Is caused by a bug that has been address on Oozie 4.3.0 that comes with HDP 2.5

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

benhadoop — Fri, 07 Oct 2016 19:47:33 GMT

Hi @Felix Albani. Thanks for your answer.

However, I recently updated to HDP 2.5 and the error still persists.

edit: I just found the mentioned issue: https://issues.apache.org/jira/browse/OOZIE-2490

HDP 2.5 only includes Oozie 4.2.0.2.5, so the solution is to wait for the next release ...

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

benhadoop — Fri, 07 Oct 2016 20:03:36 GMT

@Felix Albani Do you know how to disable ATS usage by Oozie as a workaround?

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

falbani — Fri, 07 Oct 2016 20:30:35 GMT

@Roland Simonis You are correct 2.5 comes with oozie 4.2.0.2.5. We tried to disable ATS as well on 2.4.2 with no success as we run into a different issue. Not specific to Oozie. Here is what we did: from ambari ui, went to Yarn config and set yarn.timeline-service.enabled = false. Restart all affected and test.

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

falbani — Fri, 07 Oct 2016 20:36:35 GMT

Correct, OOZIE-2490 is the error. I was told this was included in 2.5, maybe not. Sorry about the confusion.

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

benhadoop — Fri, 07 Oct 2016 20:43:47 GMT

Okay, I will try this out.

The only other workaround I can think about is getting certificates that include the IP-address. Can this work from you point of view? Or is the error created by Kerberos?

Or to disable SSL for the ATS, well effectively for complete YARN and HDFS (which is kind of bad).

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

benhadoop — Thu, 15 Dec 2016 18:52:49 GMT

Just to sum it up:

Disabling the ATS (yarn.timeline-service.enabled = false) solved the issues.

In HDP 2.5.3, the bug may be fixed, see:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/patch_hadoop.html

https://issues.apache.org/jira/browse/HADOOP-12954

or with the next version when OOZIE-2490 is included

question Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem in Archives of Support Questions (Read Only)

Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

Re: Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem