Support Questions

ask_bill_brooks · ‎07-22-2021

Failing at 38%

NAMENODE : AMBARI server

stderr: errors-440.txt

stdout: output-440.txt

2021-07-22 09:41:36,191 - Processing identities...
2021-07-22 09:41:36,222 - Creating keytab file for hdpcluster-072221@ on host myserver.com
2021-07-22 09:41:36,230 - Processing identities completed.

DATANODE : HDP CLUSTER

stdout: /var/lib/ambari-agent/data/output-438.txt

2021-07-22 09:41:34,474 - Missing keytabs:
Keytab: /etc/security/keytabs/kerberos.service_check.072221.keytab Principal: hdpcluster-072221

Command completed successfully!

This is the krb5.conf file placed in ambari server (name node) and hdpcluster 3.1.5 (datanode)

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

# output settings
[logging]
default = FILE:/tmp/krb5libs.log
kdc = FILE:/tmp/krb5kdc.log
admin_server = FILE:/tmp/kadmind.log

#Connection default configuration
[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}
udp_preference_limit= 1

[realms]
EXAMPLE.COM = {
kdc = myserver.com:88
admin_server = myserver.com
}

# domain to realm relationship (optional)
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

@Shelton

Shelton · ‎07-26-2021

@ambari275
These are the steps to follow see below

Assumptions
logged as root
clustername=test
REALM= DOMAIN.COM
Hostname = host1

logged in as root

[root@host1]#

Switch to user HDFS the HDFS superuser

[root@host1]# su - hdfs

Check the HDFS associated keytab generated

[hdfs@host1 ~]$ cd /etc/security/keytabs/
[hdfs@host1 keytabs]$ ls

Sample output

atlas.service.keytab hdfs.headless.keytab knox.service.keytab oozie.service.keytab

Now use the hdfs.headless.keytab to get the associated principal

[hdfs@host1 keytabs]$ klist -kt /etc/security/keytabs/hdfs.headless.keytab

Expected output

Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM
1 07/26/2021 00:34:03 hdfs-test@DOMAIN.COM

Grab a Kerberos ticket by using the keytab+ principal like username/pèassword to authenticate to KDC

[hdfs@host1 keytabs]$ kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-test@DOMAIN.COM

Check you no have a valid Kerberos ticket

[hdfs@host1 keytabs]$ klist

Sample output

Ticket cache: FILE:/tmp/krb5cc_1013
Default principal: hdfs-test@DOMAIN.COM
Valid starting Expires Service principal
07/26/2021 10:03:17 07/27/2021 10:03:17 krbtgt/DOMAIN.COM@DOMAIN.COM

Now you can list successfully the HDFS directories, remember to -ls it seems you forgot it in your earlier command

[hdfs@host1 keytabs]$ hdfs dfs -ls /
Found 9 items
drwxrwxrwx - yarn hadoop 0 2018-09-24 00:31 /app-logs
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:22 /apps
drwxr-xr-x - yarn hadoop 0 2018-09-24 00:12 /ats
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:12 /hdp
drwxr-xr-x - mapred hdfs 0 2018-09-24 00:12 /mapred
drwxrwxrwx - mapred hadoop 0 2018-09-24 00:12 /mr-history
drwxrwxrwx - spark hadoop 0 2021-07-26 10:04 /spark2-history
drwxrwxrwx - hdfs hdfs 0 2021-07-26 00:57 /tmp
drwxr-xr-x - hdfs hdfs 0 2018-09-24 00:23 /user

Voila happy hadooping and remember to accept the best response so other users could reference it

View solution in original post

Shelton · ‎07-26-2021

@ambari275

Great please accept the answer so the thread can be closed and referenced byother users
Happy hadooping !!!

enirys · ‎07-26-2021

hi @ambari275

i'm having same issue, which configuration solved your problem ?

ambari275 · ‎07-26-2021

what's the issue . Can you please explain at bit more .

enirys · ‎07-26-2021

I'm not able to regenerate keytabs from ambari webui, and having similar error message as you

26 Jul 2021 11:13:16,110  INFO [qtp-ambari-agent-207] HeartBeatHandler:292 - HeartBeatHandler.sendCommands: sending ExecutionCommand for host cnode43.26f5de01-5e40-4d8a-98bd-a4353b7bf5e3.datalake, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 3993-4, task ID 50394
26 Jul 2021 11:13:16,111  INFO [qtp-ambari-agent-207] HeartBeatHandler:298 - SET_KEYTAB called

26 Jul 2021 11:13:16,112  WARN [qtp-ambari-agent-207] AgentResource:136 - Error in HeartBeat
org.apache.ambari.server.AmbariException: Could not inject keytab into command
        at org.apache.ambari.server.agent.HeartBeatHandler.sendCommands(HeartBeatHandler.java:302)
        at org.apache.ambari.server.agent.HeartBeatHandler.handleHeartBeat(HeartBeatHandler.java:258)
        at org.apache.ambari.server.agent.rest.AgentResource.heartbeat(AgentResource.java:130)
        at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
        at org.apache.ambari.server.security.SecurityFilter.doFilter(SecurityFilter.java:67)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1045)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ambari.server.AmbariException: Could not inject keytabs to enable kerberos
        at org.apache.ambari.server.agent.HeartBeatHandler.injectKeytab(HeartBeatHandler.java:646)
        at org.apache.ambari.server.agent.HeartBeatHandler.sendCommands(HeartBeatHandler.java:300)
        ... 49 more
26 Jul 2021 11:13:16,113 ERROR [qtp-ambari-agent-207] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException
        at org.apache.ambari.server.agent.rest.AgentResource.heartbeat(AgentResource.java:137)
        at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
        at org.apache.ambari.server.security.SecurityFilter.doFilter(SecurityFilter.java:67)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:984)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1045)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:236)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:745)

@Sheltonany advise please ?

Shelton · ‎07-26-2021

@enirys

As suggested we need more details and there is no silver bullet a piece of advance from experience it's better you open a new thread and give as much details as possible.

OS
HDP version
Ambari
Mit or AD kerberos
Documented steps or official document reference
Your Kerberos config krb5.conf,
kdc.conf
kadm5.acl
Hosts files
Node number [Single or Multi node]

Just any information that will reduce the too many exchange of posts but gives members the info needed to help.

Cheers

cjervis · ‎09-24-2021

For reference, I resolve exactly the same issue by starting ambari-agent on the Ambari server node:

# ambari-agent start

IIRC, ambari-agent is not required to be running to enable Kerberos with Ambari 2.7.4. Don't know why it's required on 2.7.5.

------

Some further DEBUG info in ambari-server.log:

1. the keytabs were generated on ambari-server tmp directory /var/lib/ambari-server/data/tmp/ but failed to copy to /etc/security/keytabs/ on ambari-agent nodes:

2021-09-22 11:02:10,443 DEBUG [process-identity-task-110-thread-0] KerberosKeytabDAO:111 - Loading keytabs by principal name and host took 0ms
2021-09-22 11:02:10,443 INFO [process-identity-task-110-thread-0] CreateKeytabFilesServerAction:198 - Creating keytab file for hdp31-092221@EXAMPLE.COM on host node2.example.com
2021-09-22 11:02:10,444 DEBUG [process-identity-task-110-thread-0] CreateKeytabFilesServerAction:325 - Creating keytab for hdp31-092221@EXAMPLE.COM with kvno 0
2021-09-22 11:02:10,444 INFO [process-identity-task-110-thread-0] CreateKeytabFilesServerAction:257 - Successfully created keytab file for hdp31-092221@EXAMPLE.COM at /var/lib/ambari-server/data/tmp/.ambari_1632322925100-0.d/node2.example.com/6988997fbb62486a193d1de07e235fd23ffad1eb2b68837a6167409156b44444

2. "Could not inject keytab into command" error then encountered due to "Missing keytabs" on ambari-agent nodes:

......
2021-09-22 11:02:11,406  INFO [ambari-action-scheduler] AgentCommandsPublisher:124 - AgentCommandsPublisher.sendCommands: sending ExecutionCommand for host node1.example.com, role KERBEROS_CLIENT, roleCommand CUSTOM_COMMAND, and command ID 6-4, task ID 111
2021-09-22 11:02:11,406  INFO [ambari-action-scheduler] AgentCommandsPublisher:130 - SET_KEYTAB called
2021-09-22 11:02:11,412 DEBUG [ambari-action-scheduler] ActionScheduler:575 - Scheduler finished work.
2021-09-22 11:02:11,413  WARN [ambari-action-scheduler] ActionScheduler:353 - Exception received
org.apache.ambari.server.AmbariException: Could not inject keytab into command
        at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:134)
        at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.sendAgentCommand(AgentCommandsPublisher.java:92)
        at org.apache.ambari.server.actionmanager.ActionScheduler.doWork(ActionScheduler.java:557)
        at org.apache.ambari.server.actionmanager.ActionScheduler.run(ActionScheduler.java:347)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ambari.server.AmbariException: Could not inject keytabs to enable kerberos
        at org.apache.ambari.server.events.publishers.AgentCommandsPublisher$KerberosCommandParameterProcessor.process(AgentCommandsPublisher.java:261)
        at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.injectKeytab(AgentCommandsPublisher.java:184)
        at org.apache.ambari.server.events.publishers.AgentCommandsPublisher.populateExecutionCommandsClusters(AgentCommandsPublisher.java:132)
        ... 4 more

Cloudera Community

Support Questions

Kerberos issue while setting up with Ambari 2.7.5