Hello, please help me. env   jdk8 + jce   HDP 3.1.5, ambari 3.7.5 Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown: ---------------------- ambari-server.log -------------------- 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 4 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 3328 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 1024 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 3072 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 3072 2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for am - cluster[amMemory]: 3072 2022-09-26 19:02:31,601 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service AMBARI_METRICS was loaded 2022-09-26 19:02:31,604 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service HDFS was loaded 2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service SMARTSENSE was loaded 2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service ZOOKEEPER was loaded 2022-09-26 19:02:31,621 INFO HDP31HDFSServiceAdvisor getServiceConfigurationRecommendations: - Class: HDP31HDFSServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations. 2022-09-26 19:02:31,623 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Recommending Service Configurations. 2022-09-26 19:02:31,624 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Total Available Ram: 13312 2022-09-26 19:02:31,625 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS nameservices: None 2022-09-26 19:02:31,626 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Updating HDFS mount properties. 2022-09-26 19:02:31,628 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Data Dirs: [u'/data/bigdata/hdp/hadoop/hdfs/data'] 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Datanode recommended reserved size: 36805060 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HDFS service. 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK2 service. 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for YARN service. 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HIVE service. 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for OOZIE service. 2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for FALCON service. 2022-09-26 19:02:31,630 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK service. 2022-09-26 19:02:31,630 INFO DefaultStackAdvisor recommendHadoopProxyUsers: - Updated hadoop.proxyuser.hdfs.hosts as : * 2022-09-26 19:02:31,631 INFO DefaultStackAdvisor recommendConfigurationsFromHDP26: - Not setting HDFS Repo user for Ranger. Ambari returned 'hdp0.msga.com' as HST server hostname. 2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor getServiceConfigurationRecommendations: - Class: ZookeeperServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations. 2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Class: ZookeeperServiceAdvisor, Method: recommendConfigurations. Recommending Service Configurations. 2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Setting zoo.cfg to default dataDir to /hadoop/zookeeper on the best matching mount 2022-09-26 19:02:31,656 INFO [Server Action Executor Worker 73] StackAdvisorRunner:168 - Advisor script stderr: 2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service AMBARI_METRICS to auth to local mapping [explicit] 2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_COLLECTOR to auth to local mapping 2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_MONITOR to auth to local mapping 2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service HDFS to auth to local mapping [explicit] 2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component DATANODE to auth to local mapping 2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component HDFS_CLIENT to auth to local mapping 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NAMENODE to auth to local mapping 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NFS_GATEWAY to auth to local mapping 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component SECONDARY_NAMENODE to auth to local mapping 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service KERBEROS to auth to local mapping [explicit] 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component KERBEROS_CLIENT to auth to local mapping 2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service SMARTSENSE to auth to local mapping [explicit] 2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_ANALYZER to auth to local mapping 2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_EXPLORER to auth to local mapping 2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service ZOOKEEPER to auth to local mapping [explicit] 2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping 2022-09-26 19:02:32,179 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 74 2022-09-26 19:02:32,181 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called 2022-09-26 19:02:32,196 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp1.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 75 2022-09-26 19:02:32,197 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called 2022-09-26 19:02:32,199 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp2.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 76 2022-09-26 19:02:32,200 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called 2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 35 2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 36 2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 37 2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 35 2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 37 2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 36 2022-09-26 19:02:32,718 INFO [agent-report-processor-0] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp1.msga.com 2022-09-26 19:02:32,742 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp2.msga.com 2022-09-26 19:02:32,760 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp0.msga.com 2022-09-26 19:02:33,368 INFO [Server Action Executor Worker 77] KerberosServerAction:434 - Processing identities... 2022-09-26 19:02:33,407 INFO [Server Action Executor Worker 77] KerberosServerAction:493 - Processing 3 identities concurrently... 2022-09-26 19:02:33,411 INFO [process-identity-task-77-thread-0] CreatePrincipalsServerAction:240 - Processing principal, dev_test_cluster-092622@BIGDATA 2022-09-26 19:02:33,493 INFO [Server Action Executor Worker 77] KerberosServerAction:531 - Processing identities completed. 2022-09-26 19:02:34,412 INFO [Server Action Executor Worker 78] KerberosServerAction:434 - Processing identities... 2022-09-26 19:02:34,470 INFO [Server Action Executor Worker 78] KerberosServerAction:493 - Processing 3 identities concurrently... 2022-09-26 19:02:34,480 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp2.msga.com 2022-09-26 19:02:34,727 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp2.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2 2022-09-26 19:02:34,731 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp0.msga.com 2022-09-26 19:02:34,733 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp0.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2 2022-09-26 19:02:34,736 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp1.msga.com 2022-09-26 19:02:34,738 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp1.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2 2022-09-26 19:02:34,739 INFO [Server Action Executor Worker 78] KerberosServerAction:531 - Processing identities completed. 2022-09-26 19:02:35,448 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-4, task ID 79 2022-09-26 19:02:35,449 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - SET_KEYTAB called 2022-09-26 19:02:35,495 WARN [ambari-action-scheduler] ActionScheduler:353 - Exception received org.apache.ambari.server.AmbariException: Could not inject keytab into command at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:134) at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.sendAgentCommand(AgentCommandsPublisher.java:92) at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:557) at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:347) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.ambari.server.AmbariException: Could not inject keytabs to enable kerberos at org.apache.ambari.server.events.publishers.AgentCommandsPublisher$KerberosCommandParameterProcessor.process(AgentCommandsPublisher.java:261) at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.injectKeytab(AgentCommandsPublisher.java:184) at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:132) ... 4 more --------------- end ---------------- I see files in /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d. There are no files in this /etc/security/keytabs  directory. I have reinstalled the cluster, but the Kerberos configuration still throws this exception. also I'm outside the machine to build, the same configuration content, it worked!   Please help me. What should I do?  ... View more