Created on 07-19-2018 07:35 AM - edited 09-16-2022 06:29 AM
I am trying to Kerberize HDP2.6 cluster with existing windows 2012 R2 AD.
I created keytabs and principals manually but somehow they are not working and below error is observed:
18/07/19 00:18:26 WARN ipc.Client: Couldn't setup connection for hdfs@DEV.COM to single.dev.com/192.168.194.231:8020 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
Can anyone let me know how to create Keytabs and principals ( user and Service ) in windows AD so that I can use them for kerberizing the cluster manually.
Note: Using Ambari to create Principals and Keytabs is restricted.
The issue may be with the DNS or host entries in your cluster. The Hadoop services use reverse DNS to figure out the name of the host the service is on. If reverse DNS is not working, then the Hadoop service (NameNode, in this case) attempts to get a token for an invalid principal. For example: nn/192.168.194.231@DEV.COM as opposed to nn/single.dev.com@DEV.COM. Since you probably did not create a principal in your AD/KDC for nn/192.168.194.231@DEV.COM, you are getting a "Server not found in Kerberos database" error.
It could also be the your request is to http://192.168.194.231:8020 rather than http://single.dev.com:8080. In this case, the client is asking the KDC for a ticket to access HTTP/192.168.194.231@DEV.COM rather than HTTP/single.dev.com@DEV.COM. This will also result in a "Server not found in Kerberos database" error since you probably did not create an account with the a service principal name of HTTP/192.168.194.231@DEV.COM.
On this note, maybe you are missing the servicePrincipalName values for the accounts you created. All of the service principals need to be set in the relevant account's userPrincipalName and a servicePrincipalName fields. User (or headless) principals only need to have the userPrincipalName field set.
Maybe this article would be helpful - How to create AD principal accounts using OpenLdap utilities and adding it to a keytab. You might not be using OpenLDAP tools to create the accounts, but the example LDIF in there may help with the properties that need to be set for each service account.
I hope this helps...
@Robert Levas ... many thanks for your time and in my case I think reverse DNS is not working but when I integrate AD with the same cluster with Ambari Automation it works.
I am trying to create keytabs and user/server principals manually in AD and transferring them to cluster and then trying to manage kerberos.
Could you help me on how to get rid of this reverse DNS issue?
So if it all works when Ambari manages the Kerberos identities than both forward and reverse DNS are probably working. Maybe if DNS is not really set up properly, the hosts are setup on the /etc/hosts file on each node. If you think that reverse DNS is really not working, then edit the /etc/hosts file and make sure all hostnames with their appropriate IP addresses are in there. If there is a real DNS involved, you will need to talk to your system administrator since I am not one and managing a DNS server is a mystery to me. 🙂
I assume that the keytab files you manually create and distribute are correct. You need keytab files for all user/headless and service identities and they should be distributed to the appropriate hosts and set with the appropriate access control settings. All of this information should have been available in the CSV file that you can download from Ambari during and after enabling Kerberos.
Replace CLUSTER_NAME with the name of your cluster.
If you leave out the "format=CSV" you should get the information posted as a JSON documents to the screen.
This issue is not related to above case but the below error is observed when /etc/hosts is not updated.
[root@single ~]# ambari-server start Using python /usr/bin/python Starting ambari-server WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue. Ambari Server running with administrator privileges. Running initdb: This may take up to a minute. About to start PostgreSQL Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start....................................ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. [root@single ~]# [root@single ~]#
WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue.
After updating /etc/hosts file with the below details the warning is not observed. Hence, I believe reverse DNS should not be an issue and I made sure /etc/hosts is updated with complete details and I am using a single node for this purpose.
[root@single ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost ::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost
192.168.194.233 single.dev.com single
Hence, I believe reverse DNS is not an issue, please correct me.
Below did not work 😞
[root@single tmp]# ldapadd -x -H ldap://ad01.dev.com:389 -D "firstname.lastname@example.org" -W -f add_user_orig.ldif Enter LDAP Password: adding new entry "CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com" modifying entry "CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com" ldap_modify: Server is unwilling to perform (53) additional info: 0000001F: SvcErr: DSID-031A12D2, problem 5003 (WILL_NOT_PERFORM), data 0
cat add_user_orig.ldif dn: CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com changetype: add objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user distinguishedName: CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com cn: HTTP/single.dev.com userAccountControl: 514 accountExpires: 0 userPrincipalName: HTTP/single.dev.com@DEV.COM servicePrincipalName: HTTP/single.dev.com@DEV.COM dn: CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com changetype: modify replace: unicodePwd unicodePwd::IgBoAGEAZABvAG8AcABSAG8AYwBrAHMAMQAyADMAIQAiAA== dn: CN=HTTP/single.dev.com,OU=SingleDomain,DC=dev,DC=com changetype: modify replace: userAccountControl userAccountControl: 66048 [root@single tmp]#
The reason the ldapadd command failed is because you must use LDAPS rather than LDAP to perform this operation. It is an Active Directory requirement that when setting a password on an account the connection must be secure.
If you use LDAPS, the ldapadd command will want to verify the SSL certificate coming from the AD. To bypass this check, edit the /etc/openldap/ldap.conf file and update or add the following line:
@Robert Levas ... the initial error which is reported as the main subject is observed again.
Next time I used setspn command and everything went good...
Don't know where the mistake is...