Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ambari throws an exception when configuring Kerberos

avatar
New Contributor

Hello, please help me.

env

  jdk8 + jce

  HDP 3.1.5, ambari 3.7.5

Kerberos is installed successfully. When enabling Ambari Kerberos, an exception is thrown:

---------------------- ambari-server.log --------------------

2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Containers per node - cluster[containers]: 4
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Ram per containers before normalization - cluster[ramPerContainer]: 3328
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Min container size - cluster[yarnMinContainerSize]: 1024
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for map - cluster[mapMemory]: 3072
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for reduce - cluster[reduceMemory]: 3072
2022-09-26 19:02:31,600 INFO DefaultStackAdvisor getConfigurationClusterSummary: - Available memory for am - cluster[amMemory]: 3072
2022-09-26 19:02:31,601 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service AMBARI_METRICS was loaded
2022-09-26 19:02:31,604 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service HDFS was loaded
2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service SMARTSENSE was loaded
2022-09-26 19:02:31,605 INFO DefaultStackAdvisor instantiateServiceAdvisor: - ServiceAdvisor implementation for service ZOOKEEPER was loaded
2022-09-26 19:02:31,621 INFO HDP31HDFSServiceAdvisor getServiceConfigurationRecommendations: - Class: HDP31HDFSServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations.
2022-09-26 19:02:31,623 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Recommending Service Configurations.
2022-09-26 19:02:31,624 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Total Available Ram: 13312
2022-09-26 19:02:31,625 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS nameservices: None
2022-09-26 19:02:31,626 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. Updating HDFS mount properties.
2022-09-26 19:02:31,628 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Data Dirs: [u'/data/bigdata/hdp/hadoop/hdfs/data']
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor recommendConfigurationsFromHDP206: - Class: HDFSRecommender, Method: recommendConfigurationsFromHDP206. HDFS Datanode recommended reserved size: 36805060
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HDFS service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK2 service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for YARN service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for HIVE service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for OOZIE service.
2022-09-26 19:02:31,629 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for FALCON service.
2022-09-26 19:02:31,630 INFO DefaultStackAdvisor _getHadoopProxyUsersForService: - Calculating Hadoop Proxy User recommendations for SPARK service.
2022-09-26 19:02:31,630 INFO DefaultStackAdvisor recommendHadoopProxyUsers: - Updated hadoop.proxyuser.hdfs.hosts as : *
2022-09-26 19:02:31,631 INFO DefaultStackAdvisor recommendConfigurationsFromHDP26: - Not setting HDFS Repo user for Ranger.
Ambari returned 'hdp0.msga.com' as HST server hostname.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor getServiceConfigurationRecommendations: - Class: ZookeeperServiceAdvisor, Method: getServiceConfigurationRecommendations. Recommending Service Configurations.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Class: ZookeeperServiceAdvisor, Method: recommendConfigurations. Recommending Service Configurations.
2022-09-26 19:02:31,632 INFO ZookeeperServiceAdvisor recommendConfigurations: - Setting zoo.cfg to default dataDir to /hadoop/zookeeper on the best matching mount
2022-09-26 19:02:31,656 INFO [Server Action Executor Worker 73] StackAdvisorRunner:168 - Advisor script stderr:
2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service AMBARI_METRICS to auth to local mapping [explicit]
2022-09-26 19:02:31,683 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_COLLECTOR to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component METRICS_MONITOR to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service HDFS to auth to local mapping [explicit]
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component DATANODE to auth to local mapping
2022-09-26 19:02:31,684 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component HDFS_CLIENT to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NAMENODE to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component NFS_GATEWAY to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component SECONDARY_NAMENODE to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service KERBEROS to auth to local mapping [explicit]
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component KERBEROS_CLIENT to auth to local mapping
2022-09-26 19:02:31,685 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service SMARTSENSE to auth to local mapping [explicit]
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_ANALYZER to auth to local mapping
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ACTIVITY_EXPLORER to auth to local mapping
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1204 - Adding identities for service ZOOKEEPER to auth to local mapping [explicit]
2022-09-26 19:02:31,686 INFO [Server Action Executor Worker 73] KerberosHelperImpl:1231 - Adding identities for component ZOOKEEPER_SERVER to auth to local mapping
2022-09-26 19:02:32,179 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 74
2022-09-26 19:02:32,181 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,196 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp1.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 75
2022-09-26 19:02:32,197 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,199 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp2.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-1, task ID 76
2022-09-26 19:02:32,200 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - CHECK_KEYTABS called
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 35
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 36
2022-09-26 19:02:32,341 INFO [agent-message-monitor-0] MessageEmitter:218 - Schedule execution command emitting, retry: 0, messageId: 37
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 35
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 37
2022-09-26 19:02:32,343 WARN [agent-message-retry-0] MessageEmitter:255 - Reschedule execution command emitting, retry: 1, messageId: 36
2022-09-26 19:02:32,718 INFO [agent-report-processor-0] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp1.msga.com
2022-09-26 19:02:32,742 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp2.msga.com
2022-09-26 19:02:32,760 INFO [agent-report-processor-1] HeartbeatProcessor:411 - Missing principal: dev_test_cluster-092622@BIGDATA for keytab: /etc/security/keytabs/kerberos.service_check.092622.keytab on host: hdp0.msga.com
2022-09-26 19:02:33,368 INFO [Server Action Executor Worker 77] KerberosServerAction:434 - Processing identities...
2022-09-26 19:02:33,407 INFO [Server Action Executor Worker 77] KerberosServerAction:493 - Processing 3 identities concurrently...
2022-09-26 19:02:33,411 INFO [process-identity-task-77-thread-0] CreatePrincipalsServerAction:240 - Processing principal, dev_test_cluster-092622@BIGDATA
2022-09-26 19:02:33,493 INFO [Server Action Executor Worker 77] KerberosServerAction:531 - Processing identities completed.
2022-09-26 19:02:34,412 INFO [Server Action Executor Worker 78] KerberosServerAction:434 - Processing identities...
2022-09-26 19:02:34,470 INFO [Server Action Executor Worker 78] KerberosServerAction:493 - Processing 3 identities concurrently...
2022-09-26 19:02:34,480 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp2.msga.com
2022-09-26 19:02:34,727 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp2.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,731 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp0.msga.com
2022-09-26 19:02:34,733 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp0.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,736 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for dev_test_cluster-092622@BIGDATA on host hdp1.msga.com
2022-09-26 19:02:34,738 INFO [process-identity-task-78-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for dev_test_cluster-092622@BIGDATA at /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d/hdp1.msga.com/9a8b8abed46fe16464135d371567be98a9a55a89fb0150750cfe390fd34412c2
2022-09-26 19:02:34,739 INFO [Server Action Executor Worker 78] KerberosServerAction:531 - Processing identities completed.
2022-09-26 19:02:35,448 INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host hdp0.msga.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 10-4, task ID 79
2022-09-26 19:02:35,449 INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - SET_KEYTAB called
2022-09-26 19:02:35,495 WARN [ambari-action-scheduler] ActionScheduler:353 - Exception received
org.apache.ambari.server.AmbariException: Could not inject keytab into command
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:134)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.sendAgentCommand(AgentCommandsPublisher.java:92)
at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:557)
at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:347)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ambari.server.AmbariException: Could not inject keytabs to enable kerberos
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher$KerberosCommandParameterProcessor.process(AgentCommandsPublisher.java:261)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.injectKeytab(AgentCommandsPublisher.java:184)
at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:132)
... 4 more

--------------- end ----------------

I see files in /var/lib/ambari-server/data/tmp/.ambari_1664190150000-0.d.

There are no files in this /etc/security/keytabs  directory.

I have reinstalled the cluster, but the Kerberos configuration still throws this exception.

also I'm outside the machine to build, the same configuration content, it worked!

 

Please help me. What should I do? 

2 ACCEPTED SOLUTIONS

avatar
Master Mentor

@MISAKIGA 
Have you had a look at this configuring Kerberos using restAPI

View solution in original post

avatar
New Contributor

thanks for your reply !!!

I solvd the problem,This /etc/security  path has only read permission, but no open write permission.
After I gave write permission, the problem was solved and Ambari was able to create keyTab files here as desired.

Although I set all 777 permissions for /etc/security/keytabs, nothing happened

View solution in original post

2 REPLIES 2

avatar
Master Mentor

@MISAKIGA 
Have you had a look at this configuring Kerberos using restAPI

avatar
New Contributor

thanks for your reply !!!

I solvd the problem,This /etc/security  path has only read permission, but no open write permission.
After I gave write permission, the problem was solved and Ambari was able to create keyTab files here as desired.

Although I set all 777 permissions for /etc/security/keytabs, nothing happened