Dockerized YARN services with Kerberos

[#1]

I have been able to run Dockerized YARN services on a kerberized HDP 3.0.1 cluster using the following service configuration. However, this requires a service principal to be created for every node in the cluster in the format user1/hostname@EXAMPLE.COM. Additionally, the keytab for each of these principals must be distributed to their respective hosts.

Is there a way around this?

{
  "name": "hello-world",
  "version": "1.0.0",
  "description": "hello world example",
  "components" :
    [
      {
        "name": "hello",
        "number_of_containers": 5,
        "artifact": {
          "id": "library/redis",
          "type": "DOCKER"
        },
        "launch_command": "",
        "resource": {
          "cpus": 1,
          "memory": "256"
        },
        "configuration": {
          "env": {
            "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE": "true"
          }
        }
      }
    ],

    "kerberos_principal": {
      "principal_name": "user1/_HOST@EXAMPLE.COM",
      "keytab": "file:///etc/security/keytabs/user1.keytab"
    }
} 

If I leave out the "kerberos_principal" section completely, I receive this error at service submission:

{"diagnostics":"Kerberos principal or keytab is missing."}

If I use a principal without the "_HOST" portion, I receive this error at service submission:

{"diagnostics":"Kerberos principal (user1@EXAMPLE.COM) does  not contain a hostname."}

If the keytab does not exist on the worker node, I receive this error in the application log:

org.apache.hadoop.service.ServiceStateException: java.io.IOException: 
SASL is configured for registry, but neither keytab/principal nor 
java.security.auth.login.config system property are specified

[#2 · Accepted solution]

You can upload the keytab from any one host to hdfs and then set "keytab" value to that path, something like "hdfs:///user/user1/user1.keytab". Note, the principal_name in that case cannot contain _HOST anymore and has to be expanded to the hostname from where you chose to upload the keytab, so something like "user1/host1.example.com@EXAMPLE.COM".

[#3]

That worked. I just uploaded one of the keytabs into hdfs:/user/user1/user1_host1.keytab and updated the "kerberos_principal" section as follows. Is there a plan to remove the hostname requirement?

Thanks, @Gour Saha!

"kerberos_principal": {
  "principal_name": "user1/host1.example.com@EXAMPLE.COM",
  "keytab": "hdfs:/user/user1/user1_host1.keytab"
}

[#4]

YARN-7787 is open to discuss the issue, but there is no clear solution.

[#5]

Do you mean will we support principal of the format "user@EXAMPLE.COM"?

[#6]

Yes, that is what I meant.

[#7]

That's because this keytab is used by YARN Service master which needs a service principal and not a user principal. Its all towards thwarting replay attacks.