Support Questions
Find answers, ask questions, and share your expertise

Oozie launched application hangs after 24 hours on unsecured cluster

New Contributor

After 24 hours of running we see the following errors repeating in our logs

 

Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (owner=zuser, renewer=oozie mr token, realUser=oozie, issueDate=1481129498872, maxDate=1481734298872, sequenceNumber=156809, masterKeyId=1332) is expired

 

It seems we are hitting dfs.namenode.delegation.token.renew-interval  however we are running on an unsecured cluster.

 

How do we enable long running oozie launched jobs to succeed?

 

yarn 2.6.0-CDH5.5.2

oozie  4.1.0-cdh5.5.2

 

Thanks!

2 REPLIES 2

New Contributor

After more investigation I can see that something else weird is happening.

 

From the original logs

 

2016-12-08 16:55:07,566 INFO [pool-13-thread-9] org.apache.hadoop.io.retry.RetryInvocationHandler: Exception while invoking getNewApplication of class

 

Caused by:

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (owner=zuser, renewer=oozie mr token, realUser=oozie, issueDate=1481129498872, maxDate=1481734298872, sequenceNumber=156809, masterKeyId=1332) is expired

 

This says that on 2016-12-08 16:55:07 we got a log message that the token issued on 

2016-12-07 16:51:38 GMT+0000 expired.

 

Why would Oozie launch container be using such an old token?

The coordinator is a long running coordinator that materializes every 12 hours.

When is the token renewed by Oozie, each time it materializes?

New Contributor

Seriously can nobody explain what's going on here? I'm seeing the same issue and all documentation/commentary seems to indicate this shouldn't be a problem on an unsecured cluster.