question Re: Ambari is not creating keytab files though it says it has created in Archives of Support Questions (Read Only)

Ambari is not creating keytab files though it says it has created

dgiri_india1989 — Tue, 03 Oct 2017 17:08:12 GMT

Hi All,

I'm facing an issue while installing a new component to already kerberized cluster.

The installation happens successfully without any issues but services do not start due to unavailability of keytab file on that host where new component is installed.

After the installation I validated that new keytab files are not created in the designated location but ambari says it has created the keytabs and distributed to that host.

Ambari : 2.5.1

HDP : 2.6.1

Re: Ambari is not creating keytab files though it says it has created

asirna — Tue, 03 Oct 2017 17:48:31 GMT

Hi @D Giri,

Can you try re-generating the keytabs and check if it works.

Ambari=>Admin=>Kerberos => Regenerate keytabs

Thanks,

Aditya

Re: Ambari is not creating keytab files though it says it has created

Shelton — Tue, 03 Oct 2017 17:53:49 GMT

@D Giri

HDP 2.6 has a new feature called Service Auto start see Ambaru UI-->admin-> Service Auto Start

Can you validate that the component status ? Or the Auto start Services status should be either enabled/disabled

Can you also check the KDC if the principals are createdCan you also check in the KDC

# kadmin.local
kadmin.local: listprincs

Are you running Ambari as root if not then that user MUST authorization to write to /var/lib/ambari-server/tmp.

Please revert

Re: Ambari is not creating keytab files though it says it has created

dgiri_india1989 — Tue, 03 Oct 2017 19:33:10 GMT

Yes I have regenerated but it didn't help. What i have seen is it didn't create new keytabs.

Re: Ambari is not creating keytab files though it says it has created

dgiri_india1989 — Tue, 03 Oct 2017 19:42:29 GMT

Thanks @Geoffrey Shelton Okot for the quick response.

Ambari is running as an user which has got sudo privileges. And auto-start services is enabled but only metrics-collector is enabled.

In the KDC i can see that it has created corresponding principals associated with the service and hostnames.

Only issue i have observed is it stopped creating keytab files and distribute it to the designated system which ambari reported successful.

I have carried out this activity some 9-10 times but all the time its ending up without creating keytab files.

Re: Ambari is not creating keytab files though it says it has created

Shelton — Thu, 05 Oct 2017 01:47:03 GMT

@D Giri

Did you by chance download the CSV file with the keytabs for manual creation?

There is an option to ONLY regenerate keytabs for missing hosts and components !!

Did you correctly key in the user/passowrd in the Ambari-Kerberos wizard? Could you briefly describe your cluster setup? Master/slave and where the KDC is installed?

Make sure the [realms] and [domain_realms] entries in /etc/krb5.conf is correct. Validate the contents of these 2 files /var/kerberos/krb5kdc/kdc.conf , /var/kerberos/krb5kdc/kadm5.acl

Can you share the contents of the above file don't forget to scramble site specific information

Re: Ambari is not creating keytab files though it says it has created

dgiri_india1989 — Fri, 06 Oct 2017 18:49:32 GMT

Hi @Geoffrey Shelton Okot,

I have checked the content and everything looks good and we are using same krb config files across different clusters. I dont see any discrepencies with the kerberos.

Re: Ambari is not creating keytab files though it says it has created

Shelton — Fri, 06 Oct 2017 20:05:52 GMT

@D Giri

Can you descript your cluster setup (master, Slave and Edge nodes)

Okay what are the new components you are trying to install?

You could be checking for the keytabs on the wrong host, can you rerun the below command

# kadmin.local 
Authenticating as principal root/admin@REALM with password. 
kadmin.local: listprincs

All the principals created should be visible in the KD database. If the principal for the component is present take note of the host and try to locate the keytabs in the below location of that node

/etc/security/keytabs

Please let me know

Re: Ambari is not creating keytab files though it says it has created

narendrakumar — Tue, 10 Oct 2017 17:20:38 GMT

@D Giri,

Can you post the output of "ls /etc/security/keytabs" here. Along with the component that is part of cluster and fails to start ?

My suspect is that we should not put anything in "Principal Suffix" parameter filed when the keytab is created for any service. As, that adds cluster name into the keytab principle where as the service only looks by the username of respective service.

Re: Ambari is not creating keytab files though it says it has created

dgiri_india1989 — Thu, 09 Nov 2017 23:01:03 GMT

Apologies @nkumar for the delay in response.

The issue is related to Ambari which behaves differently after disabling and re-enabling the kerberos.

Issue got fixed after making changes to ambari with the help of Hortonworks Support using below REST calls.

curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS

curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS/components/KERBEROS_CLIENT

curl -s -u test:test http://ambari-server:8080/api/v1/hosts|grep host_name| sed -n 's/.*"host_name" : "$[^\"]*$".*/\1/p'>hostcluster.txt

for i in `cat hostcluster.txt`; do curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/hosts/$i/host_components/KERBEROS_CLIENT; done

curl -u test:test -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state":"INSTALLED"}}' http://ambari-server:8080/api/v1/clusters/MyClusterName/host_components?HostRoles/state=INIT

curl -H "X-Requested-By:ambari" -u test:test -i -X PUT -d @./payload.json http://ambari-server:8080/api/v1/clusters/MyClusterName

Re: Ambari is not creating keytab files though it says it has created

avinashmv442 — Sat, 10 Jul 2021 12:38:55 GMT

Hi @dgiri_india1989

Could you please share more details for this issue about how you are able to fix this.

We are also facing similar issue with Ranger KMS service.

RangerKMS Principal is created in AD KDC, Also Keytab creation is success according to Ambari Server log, but it's not distributed to RangerKMS service hosted node. Due to this service is not starting up.

Thank you.