Ambari throws an exception when configuring Kerberos

MISAKIGA — Mon, 26 Sep 2022 14:30:52 GMT

Hello, please help me.

env

jdk8 + jce

HDP 3.1.5, ambari 3.7.5

Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown:

---------------------- ambari-server.log --------------------

2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 4
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 3328
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 1024
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 3072
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 3072
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for am - cluster[amMemory]: 3072
2022-09-26 19:02:31,601 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service AMBARI_METRICS was loaded
2022-09-26 19:02:31,604 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service HDFS was loaded
2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service SMARTSENSE was loaded
2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service ZOOKEEPER was loaded
2022-09-26 19:02:31,621 INFO HDP31HDFSServiceAdvisor getServiceConfigurationRecommendations: - Class: HDP31HDFSServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations.
2022-09-26 19:02:31,623 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Recommending Service Configurations.
2022-09-26 19:02:31,624 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Total Available Ram: 13312
2022-09-26 19:02:31,625 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS nameservices: None
2022-09-26 19:02:31,626 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Updating HDFS mount properties.
2022-09-26 19:02:31,628 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Data Dirs: [u'/data/bigdata/hdp/hadoop/hdfs/data']
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Datanode recommended reserved size: 36805060
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HDFS service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK2 service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for YARN service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HIVE service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for OOZIE service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for FALCON service.
2022-09-26 19:02:31,630 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK service.
2022-09-26 19:02:31,630 INFO DefaultStackAdvisor recommendHadoopProxyUsers: - Updated hadoop.proxyuser.hdfs.hosts as : *
2022-09-26 19:02:31,631 INFO DefaultStackAdvisor recommendConfigurationsFromHDP26: - Not setting HDFS Repo user for Ranger.
Ambari returned 'hdp0.msga.com' as HST server hostname.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor getServiceConfigurationRecommendations: - Class: ZookeeperServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Class: ZookeeperServiceAdvisor, Method: recommendConfigurations. Recommending Service Configurations.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Setting zoo.cfg to default dataDir to /hadoop/zookeeper on the best matching mount
2022-09-26 19:02:31,656 INFO [Server Action Executor Worker 73] StackAdvisorRunner:168 - Advisor script stderr:
2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service AMBARI_METRICS to auth to local mapping [explicit]
2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_COLLECTOR to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_MONITOR to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service HDFS to auth to local mapping [explicit]
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component DATANODE to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component HDFS_CLIENT to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NAMENODE to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NFS_GATEWAY to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component SECONDARY_NAMENODE to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service KERBEROS to auth to local mapping [explicit]
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component KERBEROS_CLIENT to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service SMARTSENSE to auth to local mapping [explicit]
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_ANALYZER to auth to local mapping
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_EXPLORER to auth to local mapping
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service ZOOKEEPER to auth to local mapping [explicit]
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping
2022-09-26 19:02:32,179 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 74
2022-09-26 19:02:32,181 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,196 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp1.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 75
2022-09-26 19:02:32,197 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,199 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp2.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 76
2022-09-26 19:02:32,200 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 35
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 36
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 37
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 35
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 37
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 36
2022-09-26 19:02:32,718 INFO [agent-report-processor-0] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp1.msga.com
2022-09-26 19:02:32,742 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp2.msga.com
2022-09-26 19:02:32,760 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp0.msga.com
2022-09-26 19:02:33,368 INFO [Server Action Executor Worker 77] KerberosServerAction:434 - Processing identities...
2022-09-26 19:02:33,407 INFO [Server Action Executor Worker 77] KerberosServerAction:493 - Processing 3 identities concurrently...
2022-09-26 19:02:33,411 INFO [process-identity-task-77-thread-0] CreatePrincipalsServerAction:240 - Processing principal, dev_test_cluster-092622@BIGDATA
2022-09-26 19:02:33,493 INFO [Server Action Executor Worker 77] KerberosServerAction:531 - Processing identities completed.
2022-09-26 19:02:34,412 INFO [Server Action Executor Worker 78] KerberosServerAction:434 - Processing identities...
2022-09-26 19:02:34,470 INFO [Server Action Executor Worker 78] KerberosServerAction:493 - Processing 3 identities concurrently...
2022-09-26 19:02:34,480 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp2.msga.com
2022-09-26 19:02:34,727 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp2.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,731 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp0.msga.com
2022-09-26 19:02:34,733 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp0.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,736 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp1.msga.com
2022-09-26 19:02:34,738 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp1.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,739 INFO [Server Action Executor Worker 78] KerberosServerAction:531 - Processing identities completed.
2022-09-26 19:02:35,448 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-4, task ID 79
2022-09-26 19:02:35,449 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - SET_KEYTAB called
2022-09-26 19:02:35,495 WARN [ambari-action-scheduler] ActionScheduler:353 - Exception received
org.apache.ambari.server.AmbariException: Could not inject keytab into command
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:134)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.sendAgentCommand(AgentCommandsPublisher.java:92)
at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:557)
at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:347)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ambari.server.AmbariException: Could not inject keytabs to enable kerberos
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher$KerberosCommandParameterProcessor.process(AgentCommandsPublisher.java:261)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.injectKeytab(AgentCommandsPublisher.java:184)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:132)
... 4 more

--------------- end ----------------

I see files in /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d.

There are no files in this /etc/security/keytabs directory.

I have reinstalled the cluster, but the Kerberos configuration still throws this exception.

also I'm outside the machine to build, the same configuration content, it worked!

Please help me. What should I do?

Re: Ambari throws an exception when configuring Kerberos

Shelton — Tue, 27 Sep 2022 19:49:42 GMT

@MISAKIGA
Have you had a look at this configuring Kerberos using restAPI

Re: Ambari throws an exception when configuring Kerberos

MISAKIGA — Fri, 30 Sep 2022 09:42:06 GMT

thanks for your reply !!!

I solvd the problem，This /etc/security path has only read permission, but no open write permission.
After I gave write permission, the problem was solved and Ambari was able to create keyTab files here as desired.

Although I set all 777 permissions for /etc/security/keytabs, nothing happened

question Ambari throws an exception when configuring Kerberos in Support Questions

Ambari throws an exception when configuring Kerberos

Re: Ambari throws an exception when configuring Kerberos

Re: Ambari throws an exception when configuring Kerberos