CSV file is not getting generated while enabling Kerberos authentication in Apache Ambari

[#1]

@Gulshad Ansari As mentioned by you, I reran the commands and below is the output:-

[#2]

Apart from this, i can see that while i am trying to download the CSV file from Ambari UI, keytab details for all the services are not showing in the ambari ui. Should i create the keytabs for all the services manually before downloading the CSV file?

[#3]

Principal & keytabs will be created automatically by ambari in step7. It looks like a browser issue.

You can go ahead and enable the kerberos.

You will be able to download csv after setting up the kerberos.

$ curl -H "Content-Type: application/text" -H 'X-Requested-By: ambari' -u admin:admin -X GET "http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTERSNAME/kerberos_identities?fields=*&format=csv" > kerberos.csv

[#4]

@Gulshad Ansari

After enabling the kerberos in cluster, HDFS and Zookeeper services are not getting started. Please find the attached log here:- HDFS_log.txt Zookeeper_log.txt

And while downloading the CSV file, it is giving me the below output:-

[#5]

@Gulshad Ansari @Jay Kumar SenSharma

After enabling the kerberos authentication, I am still getting the attached error in the log. In zookeeper log the error is:-

File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call    raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-edtwinuat@MTN.CO.ZA;' returned 1. kinit: Client 'ambari-qa-edtwinuat@MTN.CO.ZA' not found in Kerberos database while getting initial credentials

It seems that the keytabs are not properly setup in the cluster. Do i need to manually set these keytabs up?

Please check the attached log for zookeeper and HDFS and let me know the solution?

[#6]

@Gulshad Ansari @Jay Kumar SenSharma

As the services are not running now, i disabled the kerberos and tried to re-enable it. Now i am getting the below error while re-enabling it. Please note that before kerberos enablement, HDFS was running normally.

Error message: org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Failed to create the temporary data directory.

[#7]

@Pritam Konar

As we see the error:

Error message: org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Failed to create the temporary data directory.

.

Ambari Server tmp directory location is following

# grep 'server.tmp.dir' /etc/ambari-server/conf/ambari.properties
server.tmp.dir=/var/lib/ambari-server/data/tmp

Possible Cause:

Please check the permission on the mentioned dir. The user who is running AmbariServer must be able to write inside the "/var/lib/ambari-server/data/tmp/.ambari" during kerberos setup.

So the above error indicates that you either

1. Running your Ambari Server as Non-Root user and that non root user might not have permission for writing files inside the mentioned directory.

If that is the case then please change the ownership of that dir appropriately. *Suppose* you are running AmbariServer as 'ambari' user then try this:

    # chown -R ambari:root /var/lib/ambari-server/data/tmp
    # ls -ld /var/lib/ambari-server/data/tmp

2. Or you might be having some Disk space issue which might be causing ambari server to not be able to writing inside the mentioned dir.

Relevant code:

[1] https://github.com/apache/ambari/blob/release-2.7.3/ambari-server/src/main/java/org/apache/ambari/se...

[2] https://github.com/apache/ambari/blob/release-2.7.3/ambari-server/src/main/java/org/apache/ambari/se...

[3] https://github.com/apache/ambari/blob/release-2.7.3/ambari-server/src/main/java/org/apache/ambari/se...

[#8]

@Pritam Konar

Regarding your latest error:

File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call    raise ExecutionFailed(err_msg, code, out, err) resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-edtwinuat@MTN.CO.ZA;' returned 1. kinit: Client 'ambari-qa-edtwinuat@MTN.CO.ZA' not found in Kerberos database while getting initial credentials

.

Try to regenerate theMissing Keytabs from Ambari UI to see if it is able to generate the missing keytab and able to create the missing principal?

Ambari Web > Cluster Admin > Kerberos,click Regenerate Keytabs.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/managing-and-monitoring-ambari/content/amb_...

.

[#9]

@Jay Kumar SenSharma

I followed the steps mentioned in the above Hortonworks documentation page. But when i tried to restart the services as a part of the kerberos setup, it is failing with the same error.

Please find the attached log:-

zookeeper_kerberos.txt

[#10]

@Pritam Konar

Have you tried regenerating Keytabs from Ambari UI as mentioned in my previous update?

    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-edtwinuat@MTN.CO.ZA;' returned 1. kinit: Client 'ambari-qa-edtwinuat@MTN.CO.ZA' not found in Kerberos database while getting initial credentials

Did you notice any error while generating missing keytabs?

Ambari Web > Cluster Admin > Kerberos,        click Regenerate Keytabs.

Do you see that on the Problematic host the following file is present and has correct permission?

# ls -lart /etc/security/keytabs
# ls -lart /etc/security/keytabs/smokeuser.headless.keytab 

Are you able to get a valid kerberos ticket using that keytab on that host? Please share the output of the following commands.

# klist -kte /etc/security/keytabs/smokeuser.headless.keytab
# kinit -kt /etc/security/keytabs/smokeuser.headless.keytab  ambari-qa-edtwinuat@MTN.CO.ZA
# klist
# kvno ambari-qa-edtwinuat@MTN.CO.ZA

If the file is missing on thsi host then can you try to manually copy it from any other working node. Or try to regenerate the missing keytabs form ambari UI.

.