Oozie not passing Ambari Service checks -> YARN ATS Delegation Token problem

[#1]

Hi Community,

I am experiencing weird errors with Oozie, YARN and the Application timeline server.

Running the Ambari Service checks for Oozie, Oozie is not able to get Delegation tokens for the ATS via the resourcemanager due to this error:

in resourcemanager log:

2016-10-07 13:26:43,460 WARN  security.DelegationTokenRenewer (DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(908)) - Unable to add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.40.11.42:8190, Ident: (owner=ambari-qa, renewer=yarn, realUser=oozie, issueDate=1475839603327, maxDate=1476444403327, sequenceNumber=122, masterKeyId=102)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:475)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$800(DelegationTokenRenewer.java:78)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:904)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:881)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: HTTPS hostname wrong:  should be <10.40.11.42>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:188)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:132)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:298)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:216)
	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:405)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:387)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientRetryOpForOperateDelegationToken.run(TimelineClientImpl.java:699)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:185)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:462)
	at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:409)
	at org.apache.hadoop.yarn.security.client.TimelineDelegationTokenIdentifier$Renewer.renew(TimelineDelegationTokenIdentifier.java:81)
	at org.apache.hadoop.security.token.Token.renew(Token.java:385)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:597)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:594)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:592)
	at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:461)
	... 6 more

I believe the interesting part is:

Caused by: java.io.IOException: HTTPS hostname wrong:  should be <10.40.11.42>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)

The config is:

- HDP 2.5.0.0 and Ambari 2.4.0.1

- HTTPs is activated for both Hadoop (HDFS, YARN, ATS) and Oozie. The certificates include the hostname(s) of the server

- Kerberos is activated

- The cluster is multihomed, but this communication only happens internally

- hadoop.security.token.service.use_ip is already set to false

Do you have any idea, which config I could adjust to fix this? Or is it a bug?

[#2]

@Roland Simonis Oozie will fail with this TIMELINE_DELEGATION_TOKEN when hadoop.security.token.service.use_ip=false. This is only supported in HDP 2.5 of Oozie and not in prior versions. Is caused by a bug that has been address on Oozie 4.3.0 that comes with HDP 2.5

[#3]

Hi @Felix Albani. Thanks for your answer.

However, I recently updated to HDP 2.5 and the error still persists.

edit: I just found the mentioned issue: https://issues.apache.org/jira/browse/OOZIE-2490

HDP 2.5 only includes Oozie 4.2.0.2.5, so the solution is to wait for the next release ...

[#4]

Correct, OOZIE-2490 is the error. I was told this was included in 2.5, maybe not. Sorry about the confusion.

[#5]

@Felix Albani Do you know how to disable ATS usage by Oozie as a workaround?

[#6]

@Roland Simonis You are correct 2.5 comes with oozie 4.2.0.2.5. We tried to disable ATS as well on 2.4.2 with no success as we run into a different issue. Not specific to Oozie. Here is what we did: from ambari ui, went to Yarn config and set yarn.timeline-service.enabled = false. Restart all affected and test.

[#7]

Okay, I will try this out.

The only other workaround I can think about is getting certificates that include the IP-address. Can this work from you point of view? Or is the error created by Kerberos?

Or to disable SSL for the ATS, well effectively for complete YARN and HDFS (which is kind of bad).

[#8 · Accepted solution]

Just to sum it up:

Disabling the ATS (yarn.timeline-service.enabled = false) solved the issues.

In HDP 2.5.3, the bug may be fixed, see:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/patch_hadoop.html

https://issues.apache.org/jira/browse/HADOOP-12954

or with the next version when OOZIE-2490 is included