Member since
01-19-2017
3679
Posts
632
Kudos Received
372
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 895 | 06-04-2025 11:36 PM | |
| 1491 | 03-23-2025 05:23 AM | |
| 739 | 03-17-2025 10:18 AM | |
| 2649 | 03-05-2025 01:34 PM | |
| 1767 | 03-03-2025 01:09 PM |
05-11-2018
06:37 AM
@Mudit Kumar Hopefully, this helped. If it did, please 'Accept' and 'upvote' the answer hence closing the thread. Thank you!!
... View more
05-11-2018
04:55 AM
2 Kudos
Iy you have deployed and secured your multi-node-cluster with an MIT KDC running on a Linux box (dedicated or not), this can also be applied on a single node cluster. Below is a step by step procedure to grant a group of user(s) on the Edge node with access to services in the cluster. Assumption KDC is running KDC is created KDC user and a master password is available REALM: DEV.COM Users: user1 to user5 Edge node: for users Kerberos Admin user is root or sudoer A good solution security-wise is to copy the generated keytabs to that users' home directory. If these are local Unix users NOT Active directory then create the keytabs in e.g /tmp and later copy them to their respective home directories and make sure to change the correct permissions on the keytabs. A good practice is to ensure a node dedicated to users usually called an EDGE NODE all client software is installed here and not on the Data or Name Nodes! Change directory to tmp # cd /tmp If you have root access, no need for sudo, specify the password for user1 # sudo kadmin.local
Authenticating as principal root/admin@DEV.COM with password.
kadmin.local: addprinc user1@DEV.COM
WARNING: no policy specified for user1@DEV.COM; defaulting to no policy
Enter password for principal "user1@DEV.COM":
Re-enter password for principal "user1@DEV.COM":
Principal "user1@DEV.COM" created. Do the above step for all the new users addprinc user2@DEV.COM
addprinc user3@DEV.COM
addprinc user4@DEV.COM
addprinc user5@DEV.COM The keytabs with be generated in the current directory Generate keytab for user1 The keytab will be generated in the current directory # sudo ktutil
ktutil: addent -password -p user1@DEV.COM -k 1 -e RC4-HMAC
Password for user1@DEV.COM:
ktutil: wkt user1.keytab
ktutil: q You MUST repeat the above for all the 5 users Copy the newly created keytab to the user's home directory, in this example I have copied the keytab to /etc/security/keytabs # cp user1.keytab /etc/security/keytabs Change ownership & permission here user1 belongs to hadmin group # chown user1:hadmin user1.keytab Again do the above for all the other users. A good technical and security best practice is to copy the keytabs from the kdc to edge node respective home directories and change the ownership of the keytabs Validate the principals in this example the keytabs are in /etc/security/keytabs # klist -kt /etc/security/keytabs/user1.keytab
Keytab name: FILE:/etc/security/keytabs/user1.keytab
KVNO Timestamp Principal
----------- ------------------- ------------------------------------------------------
1 05/10/2018 10:46:27 user1@DEV.COM To ensure successful ticket attribution the newly created user should validate the principal. See example below and use it grab a ticket, the principal will be concatenated with the keytab when running the kinit # klist -kt /etc/security/keytabs/user1.keytab
Keytab name: FILE:/etc/security/keytabs/user1.keytab
KVNO Timestamp Principal
-------- ------------------------ --------------------------------------------------------
1 05/10/18 01:00:50 user1@DEV.COM
.... .................. ..............
1 05/10/18 01:00:50 user1@DEV.COM Test the new user1 should try grabbing a Kerberos ticket (keytab + principal) # kinit -kt /etc/security/keytabs/user1.keytab user1@DEV.COM The below command should show the validity of the Kerberos ticket # klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: user1@DEV.COM
Valid starting Expires Service principal
05/10/2018 10:53:48 05/11/2018 10:53:48 krbtgt/DEV.COM@DEV.COM You should be okay now access and successfully run jobs on the cluster see example Accessing Hive CLI with Kerberos ticket $ hive
2018-05-10 23:18:57 WARN [main] conf.HiveConf: HiveConf of name hive.custom-extensions.root does not exist
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Logging initialized using configuration in file:/etc/hive/2.6.2.0-205/0/hive-log4j.properties
hive> show databases;
OK
default
Time taken: 8.525 seconds, Fetched: 1 row(s) Success !! Accessing Hive without a Kerberos ticket¨ Destroy the Kerberos ticket $ kdestroy Validate the existence of absence of Kerberos ticket $ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_1001) Accessing Hive CLI should fail
... View more
Labels:
05-10-2018
09:20 PM
1 Kudo
@Mudit Kumar You have deployed and secured your multi-node-cluster with an MIT KDC running on a Linux box (dedicated or not), this can also be applied on a single node cluster. Below is a step by step procedure Assumption KDC is running KDC is created KDC user and master password is available REALM: DEV.COM Users : user1,user2,user3-user5 Edge node: for users Kerberos Admin user is root or sudoer A good solution security-wise is to copy the generated keytabs to the users'home directory. If these are local Unix users NOT Active directory then create the keytabs in e.g /tmp and later copy them to their respective home directories and make sure to change the correct permissions on the keytabs. You will notice a node dedicated to users EDGE NODE, all client software are installed here and not on the data or name nodes! Change directory to tmp # cd /tmp With root access, no need for sudo, specify the password for user1 # sudo kadmin.local
Authenticating as principal root/admin@DEV.COM with password.
kadmin.local: addprinc user1@DEV.COM
WARNING: no policy specified for user1@DEV.COM; defaulting to no policy
Enter password for principal "user1@DEV.COM":
Re-enter password for principal "user1@DEV.COM":
Principal "user1@DEV.COM" created. Do the above step for for all the other users too addprinc user2@DEV.COM
addprinc user3@DEV.COM
addprinc user4@DEV.COM
addprinc user5@DEV.COM The keytabs with be generated in the current directory Generate keytab for user1 The keytab will be generated in the current directory # sudo ktutil
ktutil: addent -password -p user1@DEV.COM -k 1 -e RC4-HMAC
Password for user1@DEV.COM:
ktutil: wkt user1.keytab
ktutil: q You MUST repeat the above for all the 5 users Copy the newly created keytab to the user's home directory, in this example I have copied the keytab to /etc/security/keytabs # cp user1.keytab /etc/security/keytabs Change ownership & permission here user1 belongs to hadmin group # chown user1:hadmin user1.keytab Again do the above for all the other users. A good technical and security best practice is to copy the keytabs from the kdc to edgenode respective home directories and change the ownership of the keytabs Validate the principals in this example the keytabs are in /etc/security/keytabs # klist -kt /etc/security/keytabs/user1.keytab
Keytab name: FILE:/etc/security/keytabs/user1.keytab
KVNO Timestamp Principal
----------- ------------------- ------------------------------------------------------
1 05/10/2018 10:46:27 user1@DEV.COM To ensure successful ticket attribution the user should validate the principal see example below and use it grab a ticket , the principal will be concatenated with the keytab when running the kinit # klist -kt /etc/security/keytabs/user1.keytab
Keytab name: FILE:/etc/security/keytabs/user1.keytab
KVNO Timestamp Principal
-------- ------------------------ --------------------------------------------------------
1 05/10/18 01:00:50 user1@DEV.COM
.... .................. ..............
1 05/10/18 01:00:50 user1@DEV.COM
Test the new user1 should try grabbing a Kerberos ticket (keytab + principal) # kinit -kt /etc/security/keytabs/user1.keytab user1@DEV.COM The below command should show the validity of the Kerberos ticket # klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: user1@DEV.COM
Valid starting Expires Service principal
05/10/2018 10:53:48 05/11/2018 10:53:48 krbtgt/DEV.COM@DEV.COM You should be okay now access and successfully run jobs on the cluster
... View more
05-10-2018
02:56 PM
@Mudit Kumar Testing for HIVE or HBASE straightforward too just similar to the previous Without Kerberos ticket From ROOT switch to user hive/hbase # su - hive or # su - hbase Check if hdfs has a ticket $ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_1013) If you see some output different from the above destroy the ticket $ kdestroy Try accessing hive or Hbase shell CLI $ hive or $hbase-shell When you press "ENTER" this should give you an error for both hive and hbase Test with kerberos for hive $ klist -ket /etc/security/keytabs/hive.keytab
Keytab name: FILE:/etc/security/keytabs/hive.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (arcfour-hmac)
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (des-cbc-md5)
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) Test with kerberos for hbase $ klist -ket /etc/security/keytabs/hbase.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (arcfour-hmac)
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (des-cbc-md5)
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) Note the difference with below command it won't give you the encryption Check principal for hive $ klist -kt /etc/security/keytabs/hive.keytab
Keytab name: FILE:/etc/security/keytabs/hive.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hive-{host_name}@{YOUR_REALM} Check principal for hbase $ klist -kt /etc/security/keytabs/hbase.keytab
Keytab name: FILE:/etc/security/keytabs/hbase.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hbase-{host_name}@{YOUR_REALM} Grab a ticket, note I switch to kinit NOT klist and I append the principal for hive-{host_name}@{YOUR_REALM} or hbase-{host_name}@{YOUR_REALM} to the keytab $kinit -kt /etc/security/keytabs/hive.keytab hive-{host_name}@{YOUR_REALM} or $kinit -kt /etc/security/keytabs/hbase.keytab hbase-{host_name}@{YOUR_REALM} Now I should have a valid ticket as shown below for either hive or hbase $ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hive-{host_name}@{YOUR_REALM}
Valid starting Expires Service principal
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx krbtgt/{YOUR_REALM}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx Now I should be able to connect to hive CLI $ hive After some time you should have the below output $ hive
WARNING: Use "yarn jar" to launch YARN applications
................
Logging initialized using configuration in file:/etc/hive/2.5.0.0-817/0/hive-log4j.properties
hive> Now you can run all the hive queries For hbase $ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hbase-{host_name}@{YOUR_REALM}
Valid starting Expires Service principal
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx krbtgt/{YOUR_REALM}@{YOUR_REALM}renew until xx/xx/xx xx:xx:xx Now I should be able to connect to hbase shell and there shouldn't be any error $ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.94.23, rf42302b28aceaab773b15f234aa8718fff7eea3c, Thursday May 10 18:54:09 UTC 2018
hbase(main):001:0> Voila, you are done. As reiterated please "Accept " and close the thread. You can open a new thread as this has become long 🙂
... View more
05-10-2018
12:33 PM
@Mudit Kumar Testing is straightforward Without Kerberos ticket From ROOT switch to user hdfs # su - hdfs Check if hdfs has a ticket $ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hdfs-London@{YOUR_REALM}
Valid starting Expires Service principal
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx krbtgt/{YOUR_REALM}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx HTTP/{host_name}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx HTTP/{host_name}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
If you see some out like above that means hdfs user was already given a ticket, destroy the ticket $ kdestroy Now try accessing hdfs directory /user home $ hdfs dfs -ls /user This should throughout an error Test with Kerberos To get a valid Kerberos ticket need to know the principal, it's the part that starts with hdfs-{xxx} $ klist -ket /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (des3-cbc-sha1)
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (aes058-cts-hmac-sha1-96)
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (arcfour-hmac)
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (des-cbc-md5)
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM} (aes256-cts-hmac-sha1-96) Note the difference with below command it won't give you the encryption $ klist -kt /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM}
1 05/10/18 22:25:31 hdfs-{host_name}@{YOUR_REALM}
Grab a ticket, note I switch to kinit NOT klist and I append the principal hdfs-{host_name}@{YOUR_REALM} to the keytab $ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-{host_name}@{YOUR_REALM} Now I should have a valid ticket as shown below $ klist
Ticket cache: FILE:/tmp/krb5cc_507
Default principal: hdfs-London@{YOUR_REALM}
Valid starting Expires Service principal
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx krbtgt/{YOUR_REALM}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx HTTP/{host_name}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
xx/xx/xx xx:xx:xx xx/xx/xx xx:xx:xx HTTP/{host_name}@{YOUR_REALM} renew until xx/xx/xx xx:xx:xx
Now I should be able to list the hdfs /user directory see the example below $ hdfs dfs -ls /
Found 11 items
drwxrwxrwx - yarn hadoop 0 2018-05-09 21:45 /app-logs
.......
.......
drwxrwxrwx - mapred hadoop 0 2018-05-14 14:19 /mr-history Voila you are done, So no service /user without a valid ticket can run any job on your cluster. So can you Accept the answer I gave by Clicking on Accept button below, That would be a great help to Community users to find the solution quickly for these kinds of errors.
... View more
05-10-2018
11:56 AM
@Erkan ŞİRİN Seeing your error above "kinit: Clock skew too great while getting initial credentials" Correct me if I am wrong I see on your sandbox date output translates to date 09/05/2018 and time 09:44 # date
Wed May 9 09:44:22 +03 2018 But on the screenshot of your Windows time attached translates to date 02/05/2018 and the time 09:44 that's is 7 days difference Please set your Windows 2012R2's date to the same date like the Sandbox its should work!! Please let me know
... View more
05-09-2018
11:33 AM
Not sure that solutio will work 🙂
... View more
05-09-2018
11:03 AM
@Subramanian Govindasamy If your ticket grantor is AD the YES replace it accordingly
... View more
05-09-2018
10:50 AM
@Subramanian Govindasamy When using MIT KDC there are 3 important files that MUST be set correctly for Kerberos to function. Their locations might vary depending on the OS. /var/kerberos/krb5kdc/kdc.conf
/var/kerberos/krb5kdc/kadm5.acl
/etc/krb5.conf I have seen a couple of issues in your krb5.conf. I have corrected it replace the {your_kdc_server} see below with your KDC FQDN Back your current krb5.conf # cp /etc/krb5.conf /etc/krb5.conf.bak The edit the /etc/krb5.conf delete the contents and replace them with the below # vi /etc/krb5.conf Paste [libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = TEST.CO
dns_lookup_realm = false
ticket_lifetime = 24h
rdns = false
default_ccache_name = KEYRING:persistent:%{uid}
[domain_realm]
test.co = TEST.CO
.test.co = TEST.CO
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[realms]
TEST.CO = {
admin_server = {your_kdc_server}
kdc = {your_kdc_server}
} /var/kerberos/krb5kdc/kdc.conf [kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88
[realms]
TEST.CO = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
} Your /var/kerberos/krb5kdc/kadm5.acl should look like this note the spacing for the last * */admin@TEST.CO * Restart the KDC daemons # service /krb5kdc start
# service kadmin start Please correct the above files so we know the Kerberos is correctly set and revert with the new error if any.
... View more