Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

kms expired ticket

Highlighted

kms expired ticket

Explorer

Running CDH 5.7.3 with Kerberos, TLS/SSL level 1, and TDE/Key Trustee KMS. Have a Key Trustee Server Cluster. Everything works fine. The kms ticket liefetime is set to 7 days 

hadoop.kms.authentication.delegation-token.max-lifetime.sec  

 

After 7 days the token expires, preventing any further work. The application is a long running process where the user has loggged out. What is the best practice for renewing the ticket?

 

Thanks

 

 

 

 

The stack after 7 days:

016-11-28 19:42:51,048 ERROR AttivioEngine [EngineServerThread-12962] - ATTIVIO-INDEX_ENGINE-41 : [index.writer-part2-ba72f394-abed-4c8d-aefd-3212c96a5b6d] Fatal error occurred while indexing 
  org.apache.hadoop.security.authentication.client.AuthenticationException - org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=systemtest, renewer=yarn, realUser=, issueDate=1479767372233, maxDate=1480372172233, sequenceNumber=320, masterKeyId=13) is expired
org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=systemtest, renewer=yarn, realUser=, issueDate=1479767372233, maxDate=1480372172233, sequenceNumber=320, masterKeyId=13) is expired
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:779)
	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
	at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
	at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1483)
	at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1468)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:451)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:956)
	at com.attivio.lucene.store.hadoop.HadoopDirectory.createOutput(HadoopDirectory.java:90)
	at org.apache.lucene.store.NRTCachingDirectory.createOutput(NRTCachingDirectory.java:156)
	at com.attivio.lucene.store.AttivioDirectory.createOutput(AttivioDirectory.java:231)
	at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
	at org.apache.lucene.codecs.lucene50.Lucene50NormsConsumer.<init>(Lucene50NormsConsumer.java:64)
	at org.apache.lucene.codecs.lucene50.Lucene50NormsFormat.normsConsumer(Lucene50NormsFormat.java:123)
	at org.apache.lucene.index.DefaultIndexingChain.writeNorms(DefaultIndexingChain.java:196)
	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:95)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:420)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:512)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:624)
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2702)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2866)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2833)
	at org.apache.lucene.index.AttivioIndexWriter.commit(AttivioIndexWriter.java:67)
	at com.attivio.lucene.index.Indexer.doCommit(Indexer.java:346)
	at com.attivio.lucene.index.DocumentIndexer.commit(DocumentIndexer.java:209)
	at com.attivio.lucene.index.RealTimeZone.commit(RealTimeZone.java:396)
	at com.attivio.lucene.index.ft.FaultTolerantZone.commit(FaultTolerantZone.java:288)
	at com.attivio.lucene.index.IndexCore.commit(IndexCore.java:729)
	at com.attivio.platform.engine.AttivioEngine.startCommit(AttivioEngine.java:1444)
	at com.attivio.platform.engine.AttivioEngine.access$1000(AttivioEngine.java:90)
	at com.attivio.platform.engine.AttivioEngine$IndexingSession.commit(AttivioEngine.java:1353)
	at com.attivio.platform.engine.AttivioEngine$IndexingSession.process(AttivioEngine.java:1121)
	at com.attivio.platform.engine.ContentRequestHandler$MessageProcessor.call(ContentRequestHandler.java:434)
	at com.attivio.platform.engine.ContentRequestHandler$DispatcherInputStream.receiveMessage(ContentRequestHandler.java:366)
	at com.attivio.platform.engine.ContentRequestHandler.handle(ContentRequestHandler.java:73)
	at com.attivio.platform.engine.EngineServer$Dispatcher.run(EngineServer.java:533)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at com.attivio.platform.engine.EngineServer$ThreadFactoryRunnable.run(EngineServer.java:603)
	at java.lang.Thread.run(Thread.java:745)
2016-11-28 19:42:51,499 WARN  ContentRequestHandler [EngineServerThread-12962] - ATTIVIO-INDEX_ENGINE-23 : [/index] Node cae77489-3dd0-4e03-b739-be440bb6b17c: Engine writer-part2-ba72f394-abed-4c8d-aefd-3212c96a5b6d offline 
2016-11-28 19:42:51,500 ERROR AieIndexLauncher [Thread-603372] - ATTIVIO-PLATFORM-24 : Uncaught thread death java.lang.ThreadGroup[name=EngineServer,maxpri=10]:Thread-603372 
  org.apache.hadoop.security.authentication.client.AuthenticationException - org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=systemtest, renewer=yarn, realUser=, issueDate=1479767372233, maxDate=1480372172233, sequenceNumber=320, masterKeyId=13) can't be found in cache
org.apache.hadoop.security.authentication.client.AuthenticationException: org.apache.hadoop.security.token.SecretManager$InvalidToken: token (kms-dt owner=systemtest, renewer=yarn, realUser=, issueDate=1479767372233, maxDate=1480372172233, sequenceNumber=320, masterKeyId=13) can't be found in cache
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:157)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:546)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:504)
	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:779)
	at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
	at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1381)
	at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1483)
	at org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1468)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:451)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:956)
	at com.attivio.lucene.store.hadoop.HadoopDirectory.createOutput(HadoopDirectory.java:90)
	at org.apache.lucene.store.NRTCachingDirectory.unCache(NRTCachingDirectory.java:249)
	at org.apache.lucene.store.NRTCachingDirectory.close(NRTCachingDirectory.java:207)
	at com.attivio.lucene.store.AttivioDirectory.close(AttivioDirectory.java:263)
	at com.attivio.lucene.index.DocumentIndexer.shutdown(DocumentIndexer.java:233)
	at com.attivio.lucene.index.RealTimeZone.shutdown(RealTimeZone.java:470)
	at com.attivio.lucene.index.ft.FaultTolerantZone.shutdown(FaultTolerantZone.java:339)
	at com.attivio.lucene.index.IndexCore.shutdown(IndexCore.java:847)
	at com.attivio.platform.engine.AttivioEngine.stopComponentInternal(AttivioEngine.java:810)
	at com.attivio.platform.engine.AttivioEngine.stopComponent(AttivioEngine.java:779)
	at com.attivio.platform.engine.AttivioEngine$ShutdownThread.run(AttivioEngine.java:745)

 

 

2 REPLIES 2

Re: kms expired ticket

Explorer

This seems to be a known issue https://community.hortonworks.com/articles/74295/unable-to-put-files-in-hdfs-encrypted-zone.html with a patch in hadoop 2.8 https://issues.apache.org/jira/browse/HADOOP-13155

 

Will Cloudera be incorporating the patch?

 

Re: kms expired ticket

Rising Star

Sorry I just saw this post now.

HADOOP-13155 is included in CDH:

CDH5.4.11
CDH5.5.5 CDH5.5.6
CDH5.7.2 CDH5.7.3 CDH5.7.4 CDH5.7.5 CDH5.7.6

CDH5.8.2 CDH5.8.3 CDH5.8.4
and 5.9.x, 5.10.x and 5.11.x

Hope that helps.

Don't have an account?
Coming from Hortonworks? Activate your account here