Ambari is not creating keytab files though it says it has created

[#1]

Hi All,

I'm facing an issue while installing a new component to already kerberized cluster.

The installation happens successfully without any issues but services do not start due to unavailability of keytab file on that host where new component is installed.

After the installation I validated that new keytab files are not created in the designated location but ambari says it has created the keytabs and distributed to that host.

Ambari : 2.5.1

HDP : 2.6.1

[#2]

Hi @D Giri,

Can you try re-generating the keytabs and check if it works.

Ambari=>Admin=>Kerberos => Regenerate keytabs

Thanks,

Aditya

[#3]

Yes I have regenerated but it didn't help. What i have seen is it didn't create new keytabs.

[#4]

@D Giri

HDP 2.6 has a new feature called Service Auto start see Ambaru UI-->admin-> Service Auto Start

Can you validate that the component status ? Or the Auto start Services status should be either enabled/disabled

Can you also check the KDC if the principals are createdCan you also check in the KDC

# kadmin.local
kadmin.local: listprincs

Are you running Ambari as root if not then that user MUST authorization to write to /var/lib/ambari-server/tmp.

Please revert

[#5]

Thanks @Geoffrey Shelton Okot for the quick response.

Ambari is running as an user which has got sudo privileges. And auto-start services is enabled but only metrics-collector is enabled.

In the KDC i can see that it has created corresponding principals associated with the service and hostnames.

Only issue i have observed is it stopped creating keytab files and distribute it to the designated system which ambari reported successful.

I have carried out this activity some 9-10 times but all the time its ending up without creating keytab files.

[#6]

@D Giri

Did you by chance download the CSV file with the keytabs for manual creation?

There is an option to ONLY regenerate keytabs for missing hosts and components !!

Did you correctly key in the user/passowrd in the Ambari-Kerberos wizard? Could you briefly describe your cluster setup? Master/slave and where the KDC is installed?

Make sure the [realms] and [domain_realms] entries in /etc/krb5.conf is correct. Validate the contents of these 2 files /var/kerberos/krb5kdc/kdc.conf , /var/kerberos/krb5kdc/kadm5.acl

Can you share the contents of the above file don't forget to scramble site specific information

[#7]

Hi @Geoffrey Shelton Okot,

I have checked the content and everything looks good and we are using same krb config files across different clusters. I dont see any discrepencies with the kerberos.

[#8]

@D Giri

Can you descript your cluster setup (master, Slave and Edge nodes)

Okay what are the new components you are trying to install?

You could be checking for the keytabs on the wrong host, can you rerun the below command

# kadmin.local 
Authenticating as principal root/admin@REALM with password. 
kadmin.local: listprincs

All the principals created should be visible in the KD database. If the principal for the component is present take note of the host and try to locate the keytabs in the below location of that node

/etc/security/keytabs

Please let me know

[#9]

@D Giri,

Can you post the output of "ls /etc/security/keytabs" here. Along with the component that is part of cluster and fails to start ?

My suspect is that we should not put anything in "Principal Suffix" parameter filed when the keytab is created for any service. As, that adds cluster name into the keytab principle where as the service only looks by the username of respective service.

[#10 · Accepted solution]

Apologies @nkumar for the delay in response.

The issue is related to Ambari which behaves differently after disabling and re-enabling the kerberos.

Issue got fixed after making changes to ambari with the help of Hortonworks Support using below REST calls.

curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS

curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/services/KERBEROS/components/KERBEROS_CLIENT

curl -s -u test:test http://ambari-server:8080/api/v1/hosts|grep host_name| sed -n 's/.*"host_name" : "\([^\"]*\)".*/\1/p'>hostcluster.txt

for i in `cat hostcluster.txt`; do curl -u test:test -H "X-Requested-By: ambari" -X POST http://ambari-server:8080/api/v1/clusters/MyClusterName/hosts/$i/host_components/KERBEROS_CLIENT; done

curl -u test:test -H 'X-Requested-By: ambari' -X PUT -d '{"HostRoles": {"state":"INSTALLED"}}' http://ambari-server:8080/api/v1/clusters/MyClusterName/host_components?HostRoles/state=INIT

curl -H "X-Requested-By:ambari" -u test:test -i -X PUT -d @./payload.json http://ambari-server:8080/api/v1/clusters/MyClusterName