Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Yarn ATS Won't Start After Enabling Kerberos

avatar
Expert Contributor

After enabling Kerberos the Yarn ATS HBase Service quits working. Following the directions to destroy the service don't work due some sort of authentication issue. As you can see in my example I clearly have a Kerberos ticket for the yarn-ats users. I've also checked the Kerberos Mapping to ensure this principal is correct. I don't know what else to check.


RULE:[1:$1@$0](yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORG)s/.*/yarn-ats/


[yarn-ats@hdp31-edge ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1001
Default principal: yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORG

Valid starting       Expires              Service principal
03/15/2019 21:49:32  03/16/2019 21:49:32  krbtgt/DEV.EXAMPLE.ORG@DEV.EXAMPLE.ORG
        renew until 03/22/2019 21:49:32
[yarn-ats@hdp31-edge ~]$ yarn app -start ats-hbase
19/03/15 21:49:41 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:41 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:41 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:41 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:42 ERROR client.ApiServiceClient: Authentication required
[yarn-ats@hdp31-edge ~]$ yarn app -stop ats-hbase
19/03/15 21:49:50 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:50 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:50 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:50 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:51 ERROR client.ApiServiceClient: Authentication required
[yarn-ats@hdp31-edge ~]$ yarn app -destroy ats-hbase
19/03/15 21:49:58 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:58 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:58 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:58 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:58 ERROR client.ApiServiceClient: Authentication required
2 ACCEPTED SOLUTIONS

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Expert Contributor

Trying the equivalent with curl does this.

curl -k --negotiate -u: -H "Content-Type: application/json" -X PUT http://hdp31-mgt1.dev.example.org:8088/app/v1/services/ats-hbase -d '{  "state": "STARTED"}'
{"diagnostics":"Kerberos principal or keytab is missing."}

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

I've figured out part of the issue. For some reason all of the yarn app -status type commands are using the yarn_jaas.conf by default which directs it to use the rm/_HOST@DOMAIN.COM keytab. If I set it to use the zookeeper_client_jaas.conf which is just a generic jaas directing at your clients kerberos cache everything works fine. This seems like a bug as the client is never going to be able to use yarn_jaas.conf.


export HADOOP_OPTS='-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_client_jaas.conf'

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login