Reply
New Contributor
Posts: 3
Registered: ‎03-26-2015

Oozie launched application hangs after 24 hours on unsecured cluster

After 24 hours of running we see the following errors repeating in our logs

 

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (owner=zuser, renewer=oozie mr token, realUser=oozie, issueDate=1481129498872, maxDate=1481734298872, sequenceNumber=156809, masterKeyId=1332) is expired

 

It seems we are hitting dfs.namenode.delegation.token.renew-interval  however we are running on an unsecured cluster.

 

How do we enable long running oozie launched jobs to succeed?

 

yarn 2.6.0-CDH5.5.2

oozie  4.1.0-cdh5.5.2

 

Thanks!

New Contributor
Posts: 3
Registered: ‎03-26-2015

Re: Oozie launched application hangs after 24 hours on unsecured cluster

After more investigation I can see that something else weird is happening.

 

From the original logs

 

2016-12-08 16:55:07,566 INFO [pool-13-thread-9] org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking getNewApplication of class

 

Caused by:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (owner=zuser, renewer=oozie mr token, realUser=oozie, issueDate=1481129498872, maxDate=1481734298872, sequenceNumber=156809, masterKeyId=1332) is expired

 

This says that on 2016-12-08 16:55:07 we got a log message that the token issued on 

2016-12-07 16:51:38 GMT+0000 expired.

 

Why would Oozie launch container be using such an old token?

The coordinator is a long running coordinator that materializes every 12 hours.

When is the token renewed by Oozie, each time it materializes?

Highlighted
New Contributor
Posts: 2
Registered: ‎02-14-2017

Re: Oozie launched application hangs after 24 hours on unsecured cluster

Seriously can nobody explain what's going on here? I'm seeing the same issue and all documentation/commentary seems to indicate this shouldn't be a problem on an unsecured cluster.

Announcements