question Kerberos on Ambari 2.6.2.2: 500 status code received on POST method for API: /api/v1/clusters/hdp265/requests in Support Questions

Kerberos on Ambari 2.6.2.2: 500 status code received on POST method for API: /api/v1/clusters/hdp265/requests

vsrikanth9 — Wed, 09 Oct 2019 16:47:57 GMT

I am trying to enable Kerberos on Ambari 2.6.2.2 on CentOS 7. Below are the errors
***********************

500 status code received on POST method for API: /api/v1/clusters/hdp265/requests
Error message: An internal system exception occurred: Failed to execute the command: Broken pipe

***********************************

Below is my krb5.conf file

nano /etc/krb5.conf

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = HADOOPSECURITY.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
# EXAMPLE.COM = {
# kdc = kerberos.example.com
# admin_server = kerberos.example.com
# }
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.p1.bigdata.com = HADOOPSECURITY.COM
p1.bigdata.com = HADOOPSECURITY.COM

************************************

nano /var/kerberos/krb5kdc/kadm5.acl
*/admin@HADOOPSECURITY.COM *

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 04:47:01 GMT

I have the same issue and followed all the instructions from this post but still no luck

Re: Kerberos on Ambari 2.6.2.2: 500 status code received on POST method for API: /api/v1/clusters/hdp265/requests

Shelton — Wed, 09 Oct 2019 19:21:04 GMT

@vsrikanth9

Your krb5.conf entry is wrong please change it to match the below

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

The restart the kdc and kadmin

# systemctl start krb5kdc.service
# systemctl start kadmin.service

That should resolve your problem

Happy hadooping

Re: Kerberos on Ambari 2.6.2.2: 500 status code received on POST method for API: /api/v1/clusters/hdp265/requests

vsrikanth9 — Wed, 09 Oct 2019 23:58:52 GMT

Here I am attaching the Config screenshot. See if I am making any mistakes.

There is no change in error even after your instructions. Same error.

Re: HDP Kerberos enable through Ambari

Shelton — Thu, 10 Oct 2019 06:11:56 GMT

@vsrikanth9

1.Your KDC part of the screenshot has an error 🙂 in the domains part just copy and paste the below as is to replace p1.bigdata.com noe the dot(.) and comma separating the names

.hadoopsecurity.com,hadoopsecurity.com

The validation passed because in reality it only test the connectivity ONLY to the KDC server

2. And then the Kadmin part the Admin principal should be the output of your

# kadmin.local

Something like admin/admin@hadoopsecurity.com or root/admin@hadoopsecurity.com

What ever you chose during the installation of Kerberos after that then launch the recreation of the keytabs and all should be okay.

Make sure the KDC server is up and running during this process.

Please revert

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 12:37:10 GMT

is this what you are saying?

Also giving what i have in kadmin.local

[root@p1 /]# kadmin.local
Authenticating as principal admin/admin@HADOOPSECURITY.COM with password.
kadmin.local: listprincs
K/M@HADOOPSECURITY.COM
admin/admin@HADOOPSECURITY.COM
kadmin/admin@HADOOPSECURITY.COM
kadmin/changepw@HADOOPSECURITY.COM
kadmin/p1.bigdata.com@HADOOPSECURITY.COM
kiprop/p1.bigdata.com@HADOOPSECURITY.COM
krbtgt/HADOOPSECURITY.COM@HADOOPSECURITY.COM
test_user@HADOOPSECURITY.COM
kadmin.local:

But still the same error. I think something I am missing

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 13:12:51 GMT

I went into logs and i see below error.

stderr:
2019-10-10 09:10:37,501 - Failed to create principal, hdp265-101019@HADOOPSECURITY.COM - Failed to create service principal for hdp265-101019@HADOOPSECURITY.COM
STDOUT: Authenticating as principal admin/admin@HADOOPSECURITY.COM with password.
Password for admin/admin@HADOOPSECURITY.COM:
Enter password for principal "hdp265-101019@HADOOPSECURITY.COM":
Re-enter password for principal "hdp265-101019@HADOOPSECURITY.COM":

STDERR: WARNING: no policy specified for hdp265-101019@HADOOPSECURITY.COM; defaulting to no policy
add_principal: Operation requires ``add'' privilege while creating "hdp265-101019@HADOOPSECURITY.COM".

stdout:
2019-10-10 09:10:37,475 - Processing identities...
2019-10-10 09:10:37,482 - Processing principal, hdp265-101019@HADOOPSECURITY.COM

Re: HDP Kerberos enable through Ambari

Shelton — Thu, 10 Oct 2019 14:19:51 GMT

@vsrikanth9

Not exactly now the REALM part was wrong again the rest are okay you substituted the wrong values here is how it's supposed to be you see the highlighted part

Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[realms]
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

Do that and let me know the KDC and Admin server are usually the same 🙂

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 16:26:10 GMT

failing at the same place with same config.

nano /etc/krb5.conf

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[realms]
HADOOPSECURITY.COM = {
kdc = p1.bigdata.com
admin_server = p1.bigdata.com
}

[domain_realm]
.hadoopsecurity.com = HADOOPSECURITY.COM
hadoopsecurity.com = HADOOPSECURITY.COM

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 16:41:35 GMT

Finally, it worked when I added admin/admin into /var/kerberos/krb5kdc/kadm5.acl file. Here I have added admin/admin and root/admin as well... just created the root user.

Re: HDP Kerberos enable through Ambari

Shelton — Thu, 10 Oct 2019 17:24:22 GMT

@vsrikanth9

Great, it worked but you should recognize even if you had modified the /var/kerberos/krb5kdc/kadm5.acl still krb5.conf was wrong and your Ambari UI was wrong so you still wouldn't have resolved it 🙂

Happy hadooping

Re: HDP Kerberos enable through Ambari

vsrikanth9 — Thu, 10 Oct 2019 18:58:12 GMT

Thanks for your help Shelton. I have one other question... trying to enable Kerberos in other server but it is saying not reachable. Using similar configuration... server name different. What would be be the issue? btw. it is on HDP 3.1(Ambari 2.7) on CentOS 7 server. What could be the reason to not able to reach the KDC?