Kerberos on Ambari 2.6.2.2: 500 status code received on POST method for API: /api/v1/clusters/hdp265/requests

[#1]

I am trying to enable Kerberos on Ambari 2.6.2.2 on CentOS 7.  Below are the errors
***********************

500 status code received on POST method for API: /api/v1/clusters/hdp265/requests
Error message: An internal system exception occurred: Failed to execute the command: Broken pipe

***********************************

Below is my krb5.conf file

nano /etc/krb5.conf

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = HADOOPSECURITY.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
# EXAMPLE.COM = {
# kdc = kerberos.example.com
# admin_server = kerberos.example.com
# }
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.p1.bigdata.com = HADOOPSECURITY.COM
p1.bigdata.com = HADOOPSECURITY.COM

************************************

nano /var/kerberos/krb5kdc/kadm5.acl
*/admin@HADOOPSECURITY.COM *

[#2]

@vsrikanth9 

Your krb5.conf entry is wrong  please change it to match the below

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

The restart the kdc and kadmin

# systemctl start krb5kdc.service
# systemctl start kadmin.service

That should resolve your problem

Happy hadooping 

[#3]

Here I am attaching the Config screenshot.  See if I am making any mistakes.

There is no change in error even after your instructions.  Same error.

[#4]

I have the same issue and followed all the instructions from this post but still no luck

[#5]

@vsrikanth9 

1.Your KDC part of the screenshot has an error 🙂 in the domains part just copy and paste the below as is to replace p1.bigdata.com noe the dot(.)  and comma separating the names

.hadoopsecurity.com,hadoopsecurity.com

The validation passed because in reality it only test the connectivity ONLY to the KDC server

2. And then the Kadmin part the Admin principal should be the output of your

# kadmin.local

Something like admin/admin@hadoopsecurity.com or root/admin@hadoopsecurity.com

What ever you chose during the installation of Kerberos after that then launch the recreation of the keytabs and all should be okay.

Make sure the KDC server is up and running during this process.

Please revert

[#6]

is this what you are saying?

Also giving what i have in kadmin.local

[root@p1 /]# kadmin.local
Authenticating as principal admin/admin@HADOOPSECURITY.COM with password.
kadmin.local: listprincs
K/M@HADOOPSECURITY.COM
admin/admin@HADOOPSECURITY.COM
kadmin/admin@HADOOPSECURITY.COM
kadmin/changepw@HADOOPSECURITY.COM
kadmin/p1.bigdata.com@HADOOPSECURITY.COM
kiprop/p1.bigdata.com@HADOOPSECURITY.COM
krbtgt/HADOOPSECURITY.COM@HADOOPSECURITY.COM
test_user@HADOOPSECURITY.COM
kadmin.local:

But still the same error.  I think something I am missing

[#7]

I went into logs and i see below error.

stderr:
2019-10-10 09:10:37,501 - Failed to create principal, hdp265-101019@HADOOPSECURITY.COM - Failed to create service principal for hdp265-101019@HADOOPSECURITY.COM
STDOUT: Authenticating as principal admin/admin@HADOOPSECURITY.COM with password.
Password for admin/admin@HADOOPSECURITY.COM:
Enter password for principal "hdp265-101019@HADOOPSECURITY.COM":
Re-enter password for principal "hdp265-101019@HADOOPSECURITY.COM":

STDERR: WARNING: no policy specified for hdp265-101019@HADOOPSECURITY.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "hdp265-101019@HADOOPSECURITY.COM".

stdout:
2019-10-10 09:10:37,475 - Processing identities...
2019-10-10 09:10:37,482 - Processing principal, hdp265-101019@HADOOPSECURITY.COM

[#8]

@vsrikanth9 

Not exactly now the REALM part was wrong again the rest are okay you substituted  the wrong values here is how it's supposed to be you  see the highlighted part 

Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = HADOOPSECURITY.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

Do that and let me know the KDC and Admin server are usually the same 🙂

[#9]

failing at the same place with same config.

nano /etc/krb5.conf

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = HADOOPSECURITY.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

[#10 · Accepted solution]

Finally, it worked when I added admin/admin into /var/kerberos/krb5kdc/kadm5.acl file.  Here I have added admin/admin and root/admin as well... just created the root user.

[#11]

@vsrikanth9 

Great, it worked but you should recognize even if you had modified the    /var/kerberos/krb5kdc/kadm5.acl still  krb5.conf was wrong and your Ambari UI  was wrong so you still wouldn't have resolved it 🙂 

Happy hadooping 

[#12]

Thanks for your help Shelton.  I have one other question... trying to enable Kerberos in other server but it is saying not reachable.  Using similar configuration... server name different.  What would be be the issue?  btw. it is on HDP 3.1(Ambari 2.7) on CentOS 7 server.  What could be the reason to not able to reach the KDC?