Yarn ATS Won't Start After Enabling Kerberos

[#1]

After enabling Kerberos the Yarn ATS HBase Service quits working. Following the directions to destroy the service don't work due some sort of authentication issue. As you can see in my example I clearly have a Kerberos ticket for the yarn-ats users. I've also checked the Kerberos Mapping to ensure this principal is correct. I don't know what else to check.

RULE:[1:$1@$0](yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORG)s/.*/yarn-ats/

[yarn-ats@hdp31-edge ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_1001
Default principal: yarn-ats-hdp31_cluster@DEV.EXAMPLE.ORG

Valid starting       Expires              Service principal
03/15/2019 21:49:32  03/16/2019 21:49:32  krbtgt/DEV.EXAMPLE.ORG@DEV.EXAMPLE.ORG
        renew until 03/22/2019 21:49:32

[yarn-ats@hdp31-edge ~]$ yarn app -start ats-hbase
19/03/15 21:49:41 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:41 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:41 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:41 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:42 ERROR client.ApiServiceClient: Authentication required

[yarn-ats@hdp31-edge ~]$ yarn app -stop ats-hbase
19/03/15 21:49:50 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:50 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:50 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:50 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:51 ERROR client.ApiServiceClient: Authentication required

[yarn-ats@hdp31-edge ~]$ yarn app -destroy ats-hbase
19/03/15 21:49:58 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:58 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:58 INFO client.RMProxy: Connecting to ResourceManager at hdp31-mgt1.dev.example.org/10.0.3.52:8050
19/03/15 21:49:58 INFO client.AHSProxy: Connecting to Application History server at hdp31-mgt1.dev.example.org/10.0.3.52:10200
19/03/15 21:49:58 ERROR client.ApiServiceClient: Authentication required

[#2 · Accepted solution]

Finally managed to delete app via curl command and Ambari recreated it after a restart. Still not able to use any of the yarn app commands while logged in with the yarn-ats keytab /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab. Ambari still complains that ATS HBase isn't up but the logs in Yarn for the hbase app look like it's started.

[#3]

Trying the equivalent with curl does this.

curl -k --negotiate -u: -H "Content-Type: application/json" -X PUT http://hdp31-mgt1.dev.example.org:8088/app/v1/services/ats-hbase -d '{  "state": "STARTED"}'

{"diagnostics":"Kerberos principal or keytab is missing."}

[#4 · Accepted solution]

Finally managed to delete app via curl command and Ambari recreated it after a restart. Still not able to use any of the yarn app commands while logged in with the yarn-ats keytab /etc/security/keytabs/yarn-ats.hbase-client.headless.keytab. Ambari still complains that ATS HBase isn't up but the logs in Yarn for the hbase app look like it's started.

[#5]

I've figured out part of the issue. For some reason all of the yarn app -status type commands are using the yarn_jaas.conf by default which directs it to use the rm/_HOST@DOMAIN.COM keytab. If I set it to use the zookeeper_client_jaas.conf which is just a generic jaas directing at your clients kerberos cache everything works fine. This seems like a bug as the client is never going to be able to use yarn_jaas.conf.

export HADOOP_OPTS='-Djava.security.auth.login.config=/etc/zookeeper/conf/zookeeper_client_jaas.conf'

[#6 · Accepted solution]

And finally typing out the answer for the fourth time since I keep getting logged out. Ambari is setting rm_security_opts in yarn-env.sh to include yarn_jaas.conf. This is incorrect and breaks the yarn app commands. Commenting out that section and restarting yarn makes everything work correctly.