Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Questions regarding adding a Java KeyStore KMS

Highlighted

Questions regarding adding a Java KeyStore KMS

New Contributor

I'm setting up a Cloudera Manager 5.11.1 cluster in our lab (my first time using Cloudera Manager) and am working through setting up data-at-rest encryption for it using a java keystore kms.  The Cloudera documentation does not really detail this activity; my question is What users do I need to specify for the 2nd step of adding the Java KeyStore KMS to my cluster?  I am prompted to enter a comma-separated list of Key Admin Users and Groups; should this be the hdfs, hbase, etc users?  Or something else?

 

Any help is appreciated, thanks!

9 REPLIES 9
Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer
Following
Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Super Guru

@Platform,

 

We see your update.  Please clarify what problem you would like to discuss.  I think that "Following" may mean that you have the same question that was originally asked, but I am not certain.  Please let us know what we can do to help.

 

Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer

Hi, 

 

Could you please help me with below issue. I have configured Java keystore Kms and Enabled Encryption zone and ran spark-shell and every thing looks good. 

 

But few days, I ran spark-shell, i am getting below error. 

 

Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2)
Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.0
/_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_75)
Type in expressions to have them evaluated.
Type :help for more information.
17/12/11 11:39:56 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0008 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992364751, maxDate=1513597164751, sequenceNumber=4, masterKeyId=2)

Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer

Same issue, with hive Job as well. Not able to launch any jobs

 

Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2)

Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Super Guru

For long running Jobs, we generally recommend using the --keytab and --principal parameters as described here:

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_yarn_long_jobs.html

 

Once the delegation token expires, it must be renewed.  If you have it configured to use a keytab, then it cannot get credentials to renew the token.

 

 

Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer

Hi,

 

Thanks for the response.. It is new cluster and i have not cretaed any keytabs. I have enabled data encryption at rest using java keystore KMS and we are running spark client mode.

 

Its is kerberos enabled cluster.

 

I am testing whether all services working or not. I tried spark-shell fewdays back without passing any principals or so. It worked perfectly.

 

Today when i tried to lauch spark-shell I have faced below issue and even i am not able to insert any data into hive table,

 

Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2)

 

 

I am able to connect to hive prompt and able to create table is default database, when i tried to insert any statements i am facing below issue.

Same issue, with hive Job as well. Not able to launch any jobs

 

Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2)

 

 

Re: Questions regarding adding a Java KeyStore KMS

Explorer

beeline -u "jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXX@SB3.1ACONOMY.COM"
scan complete in 1ms
Connecting to jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXXX@SB3.1ACONOMY.COM
Connected to: Apache Hive (version 1.1.0-cdh5.8.5)
Driver: Hive JDBC (version 1.1.0-cdh5.8.5)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.8.5 by Apache Hive

 

I am able to login to beeline with the principal, but when i trying to insert the statement i am getting below issue as mentioned.

 

0: jdbc:hive2://ovhtisb3snnc01.sb3.1aconomy.c> insert into tmp_test values ('1','xxxxxxx');
INFO : Compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','amadeus')
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), FieldSchema(name:_col1, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7); Time taken: 0.341 seconds
INFO : Executing command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','xxxxxx')
INFO : Query ID = hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7
INFO : Total jobs = 3
INFO : Launching Job 1 out of 3
INFO : Starting task [Stage-1:MAPRED] in serial mode
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : number of splits:1
INFO : Submitting tokens for job: job_1512999567949_0007
INFO : Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2)
INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:sb3nameservice, Ident: (token for dwhman: HDFS_DELEGATION_TOKEN owner=dwhman, renewer=yarn, realUser=hive/ovhtisb3snnc01.sb3.1aconomy.com@SB3.1ACONOMY.COxDate=1513682115941, sequenceNumber=36, masterKeyId=53)
INFO : Kind: HIVE_DELEGATION_TOKEN, Service: HiveServer2ImpersonationToken, Ident: 00 06 64 77 68 6d 61 6e 06 64 77 68 6d 61 6e 35 68 69 76 65 2f 6f 76 68 74 69 73 62 33 73 6e 6e 63 30 31 2e 73 62 33 2e 31 6d 40 53 42 33 2e 31 41 43 4f 4e 4f 4d 59 2e 43 4f 4d 8a 01 60 4a 6f 12 32 8a 01 60 6e 7b 96 32 04 01
INFO : Cleaning up the staging area /user/dwhman/.staging/job_1512999567949_0007
ERROR : Job Submission failed with exception 'java.io.IOException(org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-d, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2))'
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-drealUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2)

 

Highlighted

Re: Questions regarding adding a Java KeyStore KMS

Explorer

 

Here is the log from KMS

 

 

Dec 14, 11:04:46.857 AM

INFOorg.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager
Updating the current master key for generating delegation tokens
Dec 14, 11:04:46.859 AMINFOorg.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager
Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
Dec 14, 11:04:46.859 AMINFOorg.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager
Updating the current master key for generating delegation tokens
Dec 14, 11:04:46.888 AMINFOcom.sun.jersey.api.core.PackagesResourceConfig
Scanning for root resource and provider classes in the packages:
  org.apache.hadoop.crypto.key.kms.server
Dec 14, 11:04:46.937 AMINFOcom.sun.jersey.api.core.ScanningResourceConfig
Root resource classes found:
  class org.apache.hadoop.crypto.key.kms.server.KMS
Dec 14, 11:04:46.937 AMINFOcom.sun.jersey.api.core.ScanningResourceConfig
Provider classes found:
  class org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider
  class org.apache.hadoop.crypto.key.kms.server.KMSJSONReader
  class org.apache.hadoop.crypto.key.kms.server.KMSJSONWriter
Dec 14, 11:04:47.019 AMINFOcom.sun.jersey.server.impl.application.WebApplicationImpl
Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
Dec 14, 1:20:22.381 PMERRORorg.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager
ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
Dec 14, 1:20:22.382 PMINFOorg.apache.hadoop.crypto.key.kms.server.KMSWebApp
KMS Stopped
Dec 14, 1:21:32.882 PMINFOorg.apache.hadoop.crypto.key.kms.server.KMSWebApp
-------------------------------------------------------------
Don't have an account?
Coming from Hortonworks? Activate your account here