My team create an yarn application which can put entities to Yarn Timeline Server periodically. After we enable security with Kerberos on our cluster, our program still works fine. But when we change the yarn-site.xml as follows:
Our program cannot finish because of the Yarn Exceptions:
So, what should we do to fix it?
This is what I get from log at DEBUG Level:
So the reason is that our token is expired, but I don't know how to renew it.
At I notice, in Tez source code, they also use TimelineClient to put entities to Yarn Timeline Server, they do not have anything to renew the token, right?
Tokens are from client application to access Yarn infrastructure. Hadoop use the org.apache.hadoop.fs.DelegationTokenRenewer thread to handle such a job. It is the responsibility of all application to renew all tokens. Start a background thread/executor to renew the tokens at a regular interval.