Support Questions
Find answers, ask questions, and share your expertise

Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

I am trying to enable Kerberos on HDP 3.1 (Ambari 2.7.1) by "Enable Kerberos wizard". However, I am getting error at step 3. I have everything setup is the same and able to enable Kerberos on HDP 2.6.

The error message is

500 status code received on POST method for API: /api/v1/clusters/Horton44/requests

Error message: An internal system exception occurred: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_6976217133962876412cc) while initializing kadmin interface

Here is the step that I am using to enable Kerberos

  1. What type of KDC do you plan on using?
    1. Existing MIT KDC Ambari Server and cluster hosts have network access to both the KDC and KDC admin hosts.
    2. KDC administrative credentials are on-hand.
    3. The Java Cryptography Extensions (JCE) have been setup on the Ambari Server host and all hosts in the cluster.
  2. Unchecked Manage Kerberos client krb5.conf (Did not work with Checked Manage Kerberos client krb5.conf as well)

At this point, the "Test Kerberos Client" is failing and got this message

  • 500 status code received on POST method for API: /api/v1/clusters/Horton44/requests
  • Error message: An internal system exception occurred: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_6976217133962876412cc) while initializing kadmin interface

Does anyone know the problem is?

Here is the log file:

2019-01-09 10:46:24,744 INFO [ambari-client-thread-43] AgentHostDataHolder:108 - Configs update with hash 25d6257c91443c9db6c5a47138a423b1a4f8edfa7ad4f15d2b04ef6eaf81977b369328bb73609b75345c1316d9caf6d15fe63ee0eb55b11d7dc43de8f44ce35c will be sent to host 1

2019-01-09 10:46:25,121 INFO [ambari-client-thread-124] MetricsCollectorHAManager:59 - Adding collector host : horton44.test.domain.com to cluster : Horton44

2019-01-09 10:46:25,123 INFO [ambari-client-thread-124] MetricsCollectorHAClusterState:84 - Refreshing collector host, current collector host : horton44.test.domain.com

2019-01-09 10:46:25,124 INFO [ambari-client-thread-124] MetricsCollectorHAClusterState:105 - After refresh, new collector host : horton44.test.domain.com

2019-01-09 10:46:25,138 INFO [ambari-client-thread-37] ServiceResourceProvider:634 - Received a updateService request, clusterName=Horton44, serviceName=KERBEROS, request=clusterName=Horton44, serviceName=KERBEROS, desiredState=INSTALLED, credentialStoreEnabled=null, credentialStoreSupported=null

2019-01-09 10:46:25,160 INFO [ambari-client-thread-37] RoleGraph:175 - Detecting cycle graphs

2019-01-09 10:46:25,160 INFO [ambari-client-thread-37] RoleGraph:176 - Graph: (KERBEROS_CLIENT, INSTALL, 0)

2019-01-09 10:46:25,311 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1062 - Host role transitioned to a new state, serviceComponentName=KERBEROS_CLIENT, hostName=horton44.test.domain.com, oldState=INIT, currentState=INSTALLING

2019-01-09 10:46:25,320 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host horton44.test.domain.com, role KERBEROS_CLIENT, roleCommand INSTALL, and command ID 15-0, task ID 152

2019-01-09 10:46:25,515 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 0

2019-01-09 10:46:25,528 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 0

2019-01-09 10:46:27,448 INFO [agent-report-processor-0] ServiceComponentHostImpl:1062 - Host role transitioned to a new state, serviceComponentName=KERBEROS_CLIENT, hostName=horton44.test.domain.com, oldState=INSTALLING, currentState=INSTALLED

2019-01-09 10:46:29,470 INFO [ambari-client-thread-43] AmbariManagementControllerImpl:4060 - Received action execution request, clusterName=Horton44, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{HAS_RESOURCE_FILTERS=true}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :Horton44

2019-01-09 10:46:39,667 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:46:49,687 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:46:59,698 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:47:09,709 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:47:09,710 WARN [ambari-client-thread-43] MITKerberosOperationHandler:302 - Failed to execute kadmin:

Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

ExitCode: 1

STDOUT: Authenticating as principal admin/admin@MIT.TESTDOMAIN.COM with existing credentials.

STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

2019-01-09 10:47:09,710 ERROR [ambari-client-thread-43] KerberosHelperImpl:2429 - Cannot validate credentials: org.apache.ambari.server.AmbariException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

2019-01-09 10:47:09,712 ERROR [ambari-client-thread-43] AbstractResourceProvider:295 - Caught AmbariException when creating a resource

org.apache.ambari.server.AmbariException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

96667-capture.png

8 REPLIES 8

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Make sure that the Kadmin Host is the FQDN of the host where the kadmin server is (which is probably the same host as the KDC). Also make sure that the following principal exists in the KDC:

kadmin/<FQDN kadmin host>@<realm>

Also, I noticed that the default realm is MIT.SSO2.RALDEV.COM where the admin principal is for MIT.TESTDOMAIN.COM. I would expect that both use the same realm name. Maybe the admin principal is incorrect and really should be admin/admin@MIT.SSO2.RALDEV.COM, or the default realm should be MIT.TESTDOMAIN.COM.

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

I changed to MIT.TESTDOMAIN.COM to post this here, but somehow, I did not change admin/admin@MIT.SSO2.RALDEV.COM to admin/admin@MIT.TESTDOMAIN.COM.

Yes, I have Kadmin Host is the FQDN of the host when I config it.

Also, on horton44 machine, I am able to do kinit admin/admin@MIT.TESTDOMAIN.COM without problem.

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

Hello,

same problem here but with ambari 2.7.3

Any solutions ?

 

Thanks a lot

 

regards

Enrico

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Super Collaborator

From Ambari 2.7.x version, for any kerberos operation, ambari will first get the service ticket for kadmin principal using admin credentials provided at the UI prompt. And executes other operations, using this service ticket stored in a temp cache. 

 

Make sure that you have kadmin service principal on KDC host, it should be in the format kadmin/<FQDN of Kadmin Host>@<REALM>. 

 

And also confirm that this principal is allowed to to tgt auth.

kadmin: getprinc kadmin/<FQDN of Kadmin Host>

 

This should not have the flag DISALLOW_TGT_BASED in attribute section. 

 

To validate that tgt auth is working, on ambari host. 

#kinit <adminPrincipal>
#kvno kadmin/<FQDN of Kadmin Host>

 

Above command should return the kvno of the principal. If it is not then, check the krb5kdc.log for any errors related to this kadmin principal. 

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Expert Contributor

Hi @rguruvannagari ,

 

How does Ambari get the kadmin principal service ticket? Is it possible to get it by running a command in the shell?

 

I'm trying to figure out a way of logging into the kadmin shell.

 

My setup is integrated with AD as the KDC. I'm trying to kadmin -p myadminprincipal@MYREALM on the terminal but it says "kadmin: Database error! Required KADM5 principal missing while initializing kadmin interface". It makes me wonder how ambari server manages to create principals and keytabs with those same credentials.

 

I even tried to decode the ambari-agent kerberos_common.py files to see the flow and there also it seems that Ambari Server indeed uses kadmin for the same.

 

Would really appreciate any help in this regard.

Thanks,

Megh

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Mentor

@EnricoTecnet 

 

Enabling Kerberos should be easy when all the steps are followed correctly. To help you resolve the issue can you share all the steps upto the enabling in Ambari UI.  [Screenshots] from Ambari would be very useful and the below files I am assuming you are on Centos or RHEL for the files paths.

  • /var/kerberos/krb5kdc/kdc.conf
  • /var/kerberos/krb5kdc/kadm5.acl
  • /etc/krb5.conf
  • MIT or AD

The above suffice for now 

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Guru
I got exactly the same issue when setting up HDP + Ambari with Kerberos.

Resolved by changing admin_server from IP address to fully qualified domain name under realms section.

Cheers
Eric

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Mentor

@EricL 

 

That's good news but it's never stressed enough. The official documentation clearly states that and the general assumption is that before starting to deploy you are expected to have gone through  this documentation else you encounter.

IP's can change in an environment for any reason and it is always recommended to use the FQDN  

 

Kerb FQDN.PNG

 

 

 

 

 

 

 

I know how frustrating it is but at the same time lessons learnt !

Happy hadooping