I am trying to Kerberize HDP2.6 cluster with existing windows 2012 R2 AD.
I created keytabs and principals manually but somehow they are not working and below error is observed:
18/07/19 00:18:26 WARN ipc.Client: Couldn't setup connection for hdfs@DEV.COM to single.dev.com/192.168.194.231:8020 javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
Can anyone let me know how to create Keytabs and principals ( user and Service ) in windows AD so that I can use them for kerberizing the cluster manually.
Note: Using Ambari to create Principals and Keytabs is restricted.
@ Sandeep, thanks for your time and I am looking for commands to create keytabs and principals on Windows AD but not on MIT KDC.
The issue may be with the DNS or host entries in your cluster. The Hadoop services use reverse DNS to figure out the name of the host the service is on. If reverse DNS is not working, then the Hadoop service (NameNode, in this case) attempts to get a token for an invalid principal. For example: nn/192.168.194.231@DEV.COM as opposed to nn/single.dev.com@DEV.COM. Since you probably did not create a principal in your AD/KDC for nn/192.168.194.231@DEV.COM, you are getting a "Server not found in Kerberos database" error.
It could also be the your request is to http://192.168.194.231:8020 rather than http://single.dev.com:8080. In this case, the client is asking the KDC for a ticket to access HTTP/192.168.194.231@DEV.COM rather than HTTP/single.dev.com@DEV.COM. This will also result in a "Server not found in Kerberos database" error since you probably did not create an account with the a service principal name of HTTP/192.168.194.231@DEV.COM.
On this note, maybe you are missing the servicePrincipalName values for the accounts you created. All of the service principals need to be set in the relevant account's userPrincipalName and a servicePrincipalName fields. User (or headless) principals only need to have the userPrincipalName field set.
Maybe this article would be helpful - How to create AD principal accounts using OpenLdap utilities and adding it to a keytab. You might not be using OpenLDAP tools to create the accounts, but the example LDIF in there may help with the properties that need to be set for each service account.
I hope this helps...
@Robert Levas ... many thanks for your time and in my case I think reverse DNS is not working but when I integrate AD with the same cluster with Ambari Automation it works.
I am trying to create keytabs and user/server principals manually in AD and transferring them to cluster and then trying to manage kerberos.
Could you help me on how to get rid of this reverse DNS issue?
So if it all works when Ambari manages the Kerberos identities than both forward and reverse DNS are probably working. Maybe if DNS is not really set up properly, the hosts are setup on the /etc/hosts file on each node. If you think that reverse DNS is really not working, then edit the /etc/hosts file and make sure all hostnames with their appropriate IP addresses are in there. If there is a real DNS involved, you will need to talk to your system administrator since I am not one and managing a DNS server is a mystery to me. :)
I assume that the keytab files you manually create and distribute are correct. You need keytab files for all user/headless and service identities and they should be distributed to the appropriate hosts and set with the appropriate access control settings. All of this information should have been available in the CSV file that you can download from Ambari during and after enabling Kerberos.
Replace CLUSTER_NAME with the name of your cluster.
If you leave out the "format=CSV" you should get the information posted as a JSON documents to the screen.
This issue is not related to above case but the below error is observed when /etc/hosts is not updated.
[root@single ~]# ambari-server start Using python /usr/bin/python Starting ambari-server WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue. Ambari Server running with administrator privileges. Running initdb: This may take up to a minute. About to start PostgreSQL Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start....................................ERROR: Exiting with exit code -1. REASON: Ambari Server java process has stopped. Please check the logs for more information. [root@single ~]# [root@single ~]#
WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior. Please check the DNS setup and fix the issue.
After updating /etc/hosts file with the below details the warning is not observed. Hence, I believe reverse DNS should not be an issue and I made sure /etc/hosts is updated with complete details and I am using a single node for this purpose.
[root@single ~]# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost.localdomain localhost4 localhost4.localdomain4 localhost ::1 localhost.localdomain localhost.localdomain localhost6 localhost6.localdomain6 localhost
192.168.194.233 single.dev.com single
Hence, I believe reverse DNS is not an issue, please correct me.
I created all keytabs and principals and performed Kerberisation.
My main point of concern is:
How to create service principals?
Correct, I think reverse DNS is working and not an issue after fixing the /etc/hosts file.