Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

I am trying to enable Kerberos on HDP 3.1 (Ambari 2.7.1) by "Enable Kerberos wizard". However, I am getting error at step 3. I have everything setup is the same and able to enable Kerberos on HDP 2.6.

The error message is

500 status code received on POST method for API: /api/v1/clusters/Horton44/requests

Error message: An internal system exception occurred: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_6976217133962876412cc) while initializing kadmin interface

Here is the step that I am using to enable Kerberos

  1. What type of KDC do you plan on using?
    1. Existing MIT KDC Ambari Server and cluster hosts have network access to both the KDC and KDC admin hosts.
    2. KDC administrative credentials are on-hand.
    3. The Java Cryptography Extensions (JCE) have been setup on the Ambari Server host and all hosts in the cluster.
  2. Unchecked Manage Kerberos client krb5.conf (Did not work with Checked Manage Kerberos client krb5.conf as well)

At this point, the "Test Kerberos Client" is failing and got this message

  • 500 status code received on POST method for API: /api/v1/clusters/Horton44/requests
  • Error message: An internal system exception occurred: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_6976217133962876412cc) while initializing kadmin interface

Does anyone know the problem is?

Here is the log file:

2019-01-09 10:46:24,744 INFO [ambari-client-thread-43] AgentHostDataHolder:108 - Configs update with hash 25d6257c91443c9db6c5a47138a423b1a4f8edfa7ad4f15d2b04ef6eaf81977b369328bb73609b75345c1316d9caf6d15fe63ee0eb55b11d7dc43de8f44ce35c will be sent to host 1

2019-01-09 10:46:25,121 INFO [ambari-client-thread-124] MetricsCollectorHAManager:59 - Adding collector host : horton44.test.domain.com to cluster : Horton44

2019-01-09 10:46:25,123 INFO [ambari-client-thread-124] MetricsCollectorHAClusterState:84 - Refreshing collector host, current collector host : horton44.test.domain.com

2019-01-09 10:46:25,124 INFO [ambari-client-thread-124] MetricsCollectorHAClusterState:105 - After refresh, new collector host : horton44.test.domain.com

2019-01-09 10:46:25,138 INFO [ambari-client-thread-37] ServiceResourceProvider:634 - Received a updateService request, clusterName=Horton44, serviceName=KERBEROS, request=clusterName=Horton44, serviceName=KERBEROS, desiredState=INSTALLED, credentialStoreEnabled=null, credentialStoreSupported=null

2019-01-09 10:46:25,160 INFO [ambari-client-thread-37] RoleGraph:175 - Detecting cycle graphs

2019-01-09 10:46:25,160 INFO [ambari-client-thread-37] RoleGraph:176 - Graph: (KERBEROS_CLIENT, INSTALL, 0)

2019-01-09 10:46:25,311 INFO [ambari-action-scheduler] ServiceComponentHostImpl:1062 - Host role transitioned to a new state, serviceComponentName=KERBEROS_CLIENT, hostName=horton44.test.domain.com, oldState=INIT, currentState=INSTALLING

2019-01-09 10:46:25,320 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host horton44.test.domain.com, role KERBEROS_CLIENT, roleCommand INSTALL, and command ID 15-0, task ID 152

2019-01-09 10:46:25,515 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 0

2019-01-09 10:46:25,528 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 0

2019-01-09 10:46:27,448 INFO [agent-report-processor-0] ServiceComponentHostImpl:1062 - Host role transitioned to a new state, serviceComponentName=KERBEROS_CLIENT, hostName=horton44.test.domain.com, oldState=INSTALLING, currentState=INSTALLED

2019-01-09 10:46:29,470 INFO [ambari-client-thread-43] AmbariManagementControllerImpl:4060 - Received action execution request, clusterName=Horton44, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{HAS_RESOURCE_FILTERS=true}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :Horton44

2019-01-09 10:46:39,667 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:46:49,687 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:46:59,698 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:47:09,709 WARN [ambari-client-thread-43] MITKerberosOperationHandler:291 - Retrying to execute kadmin after a wait of 10 seconds : Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

2019-01-09 10:47:09,710 WARN [ambari-client-thread-43] MITKerberosOperationHandler:302 - Failed to execute kadmin:

Command: [/usr/bin/kadmin, -c, /tmp/ambari_krb_5117636388301835326cc, -s, nc-mit-kdc.sso2.raldev.com, -r, MIT.SSO2.RALDEV.COM, -q, get_principal admin/admin@MIT.TESTDOMAIN.COM]

ExitCode: 1

STDOUT: Authenticating as principal admin/admin@MIT.TESTDOMAIN.COM with existing credentials.

STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

2019-01-09 10:47:09,710 ERROR [ambari-client-thread-43] KerberosHelperImpl:2429 - Cannot validate credentials: org.apache.ambari.server.AmbariException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

2019-01-09 10:47:09,712 ERROR [ambari-client-thread-43] AbstractResourceProvider:295 - Caught AmbariException when creating a resource

org.apache.ambari.server.AmbariException: Unexpected error condition executing the kadmin command. STDERR: kadmin: Matching credential not found (filename: /tmp/ambari_krb_5117636388301835326cc) while initializing kadmin interface

96667-capture.png

5 REPLIES 5

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Make sure that the Kadmin Host is the FQDN of the host where the kadmin server is (which is probably the same host as the KDC). Also make sure that the following principal exists in the KDC:

kadmin/<FQDN kadmin host>@<realm>

Also, I noticed that the default realm is MIT.SSO2.RALDEV.COM where the admin principal is for MIT.TESTDOMAIN.COM. I would expect that both use the same realm name. Maybe the admin principal is incorrect and really should be admin/admin@MIT.SSO2.RALDEV.COM, or the default realm should be MIT.TESTDOMAIN.COM.

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

I changed to MIT.TESTDOMAIN.COM to post this here, but somehow, I did not change admin/admin@MIT.SSO2.RALDEV.COM to admin/admin@MIT.TESTDOMAIN.COM.

Yes, I have Kadmin Host is the FQDN of the host when I config it.

Also, on horton44 machine, I am able to do kinit admin/admin@MIT.TESTDOMAIN.COM without problem.

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

New Contributor

Hello,

same problem here but with ambari 2.7.3

Any solutions ?

 

Thanks a lot

 

regards

Enrico

Highlighted

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Super Collaborator

From Ambari 2.7.x version, for any kerberos operation, ambari will first get the service ticket for kadmin principal using admin credentials provided at the UI prompt. And executes other operations, using this service ticket stored in a temp cache. 

 

Make sure that you have kadmin service principal on KDC host, it should be in the format kadmin/<FQDN of Kadmin Host>@<REALM>. 

 

And also confirm that this principal is allowed to to tgt auth.

kadmin: getprinc kadmin/<FQDN of Kadmin Host>

 

This should not have the flag DISALLOW_TGT_BASED in attribute section. 

 

To validate that tgt auth is working, on ambari host. 

#kinit <adminPrincipal>
#kvno kadmin/<FQDN of Kadmin Host>

 

Above command should return the kvno of the principal. If it is not then, check the krb5kdc.log for any errors related to this kadmin principal. 

Re: Failed to enable Kerberos Services on HDP 3.1 (Ambari 2.7.1)

Mentor

@EnricoTecnet 

 

Enabling Kerberos should be easy when all the steps are followed correctly. To help you resolve the issue can you share all the steps upto the enabling in Ambari UI.  [Screenshots] from Ambari would be very useful and the below files I am assuming you are on Centos or RHEL for the files paths.

  • /var/kerberos/krb5kdc/kdc.conf
  • /var/kerberos/krb5kdc/kadm5.acl
  • /etc/krb5.conf
  • MIT or AD

The above suffice for now 

Don't have an account?
Coming from Hortonworks? Activate your account here