I was able to fix the issue. It was due to firewalls. ... View more

Hi Rajat,   Could you please help me upto setup Cross Realm trust?     Thanks, Udai ... View more

Could you please help me with the steps. If you have any Document, please let me know.     Thanks,   ... View more

  Here is the log from KMS     Dec 14, 11:04:46.857 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Updating the current master key for generating delegation tokens Dec 14, 11:04:46.859 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) Dec 14, 11:04:46.859 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Updating the current master key for generating delegation tokens Dec 14, 11:04:46.888 AM INFO com.sun.jersey.api.core.PackagesResourceConfig Scanning for root resource and provider classes in the packages: org.apache.hadoop.crypto.key.kms.server Dec 14, 11:04:46.937 AM INFO com.sun.jersey.api.core.ScanningResourceConfig Root resource classes found: class org.apache.hadoop.crypto.key.kms.server.KMS Dec 14, 11:04:46.937 AM INFO com.sun.jersey.api.core.ScanningResourceConfig Provider classes found: class org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider class org.apache.hadoop.crypto.key.kms.server.KMSJSONReader class org.apache.hadoop.crypto.key.kms.server.KMSJSONWriter Dec 14, 11:04:47.019 AM INFO com.sun.jersey.server.impl.application.WebApplicationImpl Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Dec 14, 1:20:22.381 PM ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted Dec 14, 1:20:22.382 PM INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp KMS Stopped Dec 14, 1:21:32.882 PM INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp ------------------------------------------------------------- ... View more

beeline -u "jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXX@SB3.1ACONOMY.COM" scan complete in 1ms Connecting to jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXXX@SB3.1ACONOMY.COM Connected to: Apache Hive (version 1.1.0-cdh5.8.5) Driver: Hive JDBC (version 1.1.0-cdh5.8.5) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.8.5 by Apache Hive   I am able to login to beeline with the principal, but when i trying to insert the statement i am getting below issue as mentioned.   0: jdbc:hive2://ovhtisb3snnc01.sb3.1aconomy.c> insert into tmp_test values ('1','xxxxxxx'); INFO : Compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','amadeus') INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), FieldSchema(name:_col1, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7); Time taken: 0.341 seconds INFO : Executing command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','xxxxxx') INFO : Query ID = hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7 INFO : Total jobs = 3 INFO : Launching Job 1 out of 3 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : number of splits:1 INFO : Submitting tokens for job: job_1512999567949_0007 INFO : Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2) INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:sb3nameservice, Ident: (token for dwhman: HDFS_DELEGATION_TOKEN owner=dwhman, renewer=yarn, realUser=hive/ovhtisb3snnc01.sb3.1aconomy.com@SB3.1ACONOMY.COxDate=1513682115941, sequenceNumber=36, masterKeyId=53) INFO : Kind: HIVE_DELEGATION_TOKEN, Service: HiveServer2ImpersonationToken, Ident: 00 06 64 77 68 6d 61 6e 06 64 77 68 6d 61 6e 35 68 69 76 65 2f 6f 76 68 74 69 73 62 33 73 6e 6e 63 30 31 2e 73 62 33 2e 31 6d 40 53 42 33 2e 31 41 43 4f 4e 4f 4d 59 2e 43 4f 4d 8a 01 60 4a 6f 12 32 8a 01 60 6e 7b 96 32 04 01 INFO : Cleaning up the staging area /user/dwhman/.staging/job_1512999567949_0007 ERROR : Job Submission failed with exception 'java.io.IOException(org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-d, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2))' java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-drealUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2)   ... View more

Hi,   Thanks for the response.. It is new cluster and i have not cretaed any keytabs. I have enabled data encryption at rest using java keystore KMS and we are running spark client mode.   Its is kerberos enabled cluster.   I am testing whether all services working or not. I tried spark-shell fewdays back without passing any principals or so. It worked perfectly.   Today when i tried to lauch spark-shell I have faced below issue and even i am not able to insert any data into hive table,   Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2)     I am able to connect to hive prompt and able to create table is default database, when i tried to insert any statements i am facing below issue. Same issue, with hive Job as well. Not able to launch any jobs   Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2)     ... View more

Same issue, with hive Job as well. Not able to launch any jobs   Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2) ... View more

Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_75) Type in expressions to have them evaluated. Type :help for more information. 17/12/11 11:39:56 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0008 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992364751, maxDate=1513597164751, sequenceNumber=4, masterKeyId=2) ... View more

Hi,    Could you please help me with below issue. I have configured Java keystore Kms and Enabled Encryption zone and ran spark-shell and every thing looks good.    But few days, I ran spark-shell, i am getting below error.    Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2) ... View more

Following ... View more