Member since
11-27-2017
15
Posts
0
Kudos Received
0
Solutions
02-28-2018
01:34 AM
Thanks... Following the cloudera Doc's I was able to sucessfully setup Cross-realm trust. Issue is with DNS.
... View more
02-28-2018
01:32 AM
I was able to fix the issue. It was due to firewalls.
... View more
02-28-2018
01:29 AM
Hi Team,
I have 2 clusters A and B.
i) Cluster A is Kerberos enabled and it has Encryption_zone with KMS.
ii) Cluster B is Kerberos enabled and it has Encryption_zone with KMS.
From cluster A to B I have enabled trust between 2 KDC's and its working fine. I was able to do Distcp from B cluster.
I am able to access the cluster A from cluster B and able read the data from Users Home directory.
1)But I have a requirement such way that from Cluster B I have read cluster A encrytion_zone data. I looged with Cluster A kerberos Cred's in cluster B and when i am trying to access cluster A encryption_zone i am not able to see decrypt output.
2) Through spark-shell, I read cluster B data and trying access and facing below error.
I am doing this from cluster B
scala> val txt = sc.textFile("hdfs://Exnameservice/user/Exuser/tmp/sk_stg.conf") txt: org.apache.spark.rdd.RDD[String] = hdfs:// Exnameservice /user/Exuser/tmp/sk_stg.conf MapPartitionsRDD[1] at textFile at <console>:27
scala> txt.take(3)
18/02/26 14:41:45 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, clusterB15.examaple.com): java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: " clusterB15.examaple.com /172.xx.xx.xx"; destination host is: "clusterAnamenode1.example1.com":8020;
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
Could you please someone help here.
... View more
01-31-2018
05:55 AM
Hi Harsha, Need your help in setting up cross-realm Auth. How can i contact you ? Thanks,
... View more
01-31-2018
05:54 AM
Hi Harsha, I need your help in Setting up Cross-Realm Auth between 2 secured clusters. How can i reach you? Thanks,
... View more
01-31-2018
03:19 AM
Hi Rajat, Could you please help me upto setup Cross Realm trust? Thanks, Udai
... View more
01-24-2018
05:53 AM
Could you please help me with the steps. If you have any Document, please let me know. Thanks,
... View more
01-18-2018
03:41 AM
Hi Team, Cloud you please let me know how to distcp between Insecure cluster to Secured cluster? Thanks,
... View more
Labels:
12-14-2017
06:19 AM
Here is the log from KMS Dec 14, 11:04:46.857 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Updating the current master key for generating delegation tokens Dec 14, 11:04:46.859 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) Dec 14, 11:04:46.859 AM INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager Updating the current master key for generating delegation tokens Dec 14, 11:04:46.888 AM INFO com.sun.jersey.api.core.PackagesResourceConfig Scanning for root resource and provider classes in the packages:
org.apache.hadoop.crypto.key.kms.server Dec 14, 11:04:46.937 AM INFO com.sun.jersey.api.core.ScanningResourceConfig Root resource classes found:
class org.apache.hadoop.crypto.key.kms.server.KMS Dec 14, 11:04:46.937 AM INFO com.sun.jersey.api.core.ScanningResourceConfig Provider classes found:
class org.apache.hadoop.crypto.key.kms.server.KMSExceptionsProvider
class org.apache.hadoop.crypto.key.kms.server.KMSJSONReader
class org.apache.hadoop.crypto.key.kms.server.KMSJSONWriter Dec 14, 11:04:47.019 AM INFO com.sun.jersey.server.impl.application.WebApplicationImpl Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' Dec 14, 1:20:22.381 PM ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted Dec 14, 1:20:22.382 PM INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp KMS Stopped Dec 14, 1:21:32.882 PM INFO org.apache.hadoop.crypto.key.kms.server.KMSWebApp -------------------------------------------------------------
... View more
12-12-2017
03:52 AM
beeline -u "jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXX@SB3.1ACONOMY.COM" scan complete in 1ms Connecting to jdbc:hive2://XXXXXXX:10000/default;principal=hive/XXXXXXXX@SB3.1ACONOMY.COM Connected to: Apache Hive (version 1.1.0-cdh5.8.5) Driver: Hive JDBC (version 1.1.0-cdh5.8.5) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.8.5 by Apache Hive I am able to login to beeline with the principal, but when i trying to insert the statement i am getting below issue as mentioned. 0: jdbc:hive2://ovhtisb3snnc01.sb3.1aconomy.c> insert into tmp_test values ('1','xxxxxxx'); INFO : Compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','amadeus') INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_col0, type:int, comment:null), FieldSchema(name:_col1, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7); Time taken: 0.341 seconds INFO : Executing command(queryId=hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7): insert into tmp_test values ('1','xxxxxx') INFO : Query ID = hive_20171212111515_eda7838d-97b2-4050-a851-04d52b716ae7 INFO : Total jobs = 3 INFO : Launching Job 1 out of 3 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : number of splits:1 INFO : Submitting tokens for job: job_1512999567949_0007 INFO : Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2) INFO : Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:sb3nameservice, Ident: (token for dwhman: HDFS_DELEGATION_TOKEN owner=dwhman, renewer=yarn, realUser=hive/ovhtisb3snnc01.sb3.1aconomy.com@SB3.1ACONOMY.COxDate=1513682115941, sequenceNumber=36, masterKeyId=53) INFO : Kind: HIVE_DELEGATION_TOKEN, Service: HiveServer2ImpersonationToken, Ident: 00 06 64 77 68 6d 61 6e 06 64 77 68 6d 61 6e 35 68 69 76 65 2f 6f 76 68 74 69 73 62 33 73 6e 6e 63 30 31 2e 73 62 33 2e 31 6d 40 53 42 33 2e 31 41 43 4f 4e 4f 4d 59 2e 43 4f 4d 8a 01 60 4a 6f 12 32 8a 01 60 6e 7b 96 32 04 01 INFO : Cleaning up the staging area /user/dwhman/.staging/job_1512999567949_0007 ERROR : Job Submission failed with exception 'java.io.IOException(org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-d, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2))' java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512999567949_0007 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-drealUser=hive, issueDate=1513077316273, maxDate=1513682116273, sequenceNumber=7, masterKeyId=2)
... View more
12-11-2017
09:59 AM
Hi, Thanks for the response.. It is new cluster and i have not cretaed any keytabs. I have enabled data encryption at rest using java keystore KMS and we are running spark client mode. Its is kerberos enabled cluster. I am testing whether all services working or not. I tried spark-shell fewdays back without passing any principals or so. It worked perfectly. Today when i tried to lauch spark-shell I have faced below issue and even i am not able to insert any data into hive table, Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2) I am able to connect to hive prompt and able to create table is default database, when i tried to insert any statements i am facing below issue. Same issue, with hive Job as well. Not able to launch any jobs Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2)
... View more
12-11-2017
03:45 AM
Same issue, with hive Job as well. Not able to launch any jobs Launching Job 1 out of 3 Number of reduce tasks is set to 0 since there's no reduce operator java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0009 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992649605, maxDate=1513597449605, sequenceNumber=5, masterKeyId=2)
... View more
12-11-2017
03:41 AM
Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 /_/ Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_75) Type in expressions to have them evaluated. Type :help for more information. 17/12/11 11:39:56 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1512658370314_0008 to YARN : Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512992364751, maxDate=1513597164751, sequenceNumber=4, masterKeyId=2)
... View more
12-11-2017
03:18 AM
Hi, Could you please help me with below issue. I have configured Java keystore Kms and Enabled Encryption zone and ran spark-shell and every thing looks good. But few days, I ran spark-shell, i am getting below error. Failed to renew token: Kind: kms-dt, Service: 172.16.8.160:16000, Ident: (kms-dt owner=dwhman, renewer=yarn, realUser=, issueDate=1512928102644, maxDate=1513532902644, sequenceNumber=3, masterKeyId=2)
... View more