question Re: Dockerized YARN services with Kerberos in Archives of Support Questions (Read Only)

Dockerized YARN services with Kerberos

shjelmfelt — Sat, 10 Nov 2018 08:35:42 GMT

I have been able to run Dockerized YARN services on a kerberized HDP 3.0.1 cluster using the following service configuration. However, this requires a service principal to be created for every node in the cluster in the format user1/hostname@EXAMPLE.COM. Additionally, the keytab for each of these principals must be distributed to their respective hosts.

Is there a way around this?

{
  "name": "hello-world",
  "version": "1.0.0",
  "description": "hello world example",
  "components" :
    [
      {
        "name": "hello",
        "number_of_containers": 5,
        "artifact": {
          "id": "library/redis",
          "type": "DOCKER"
        },
        "launch_command": "",
        "resource": {
          "cpus": 1,
          "memory": "256"
        },
        "configuration": {
          "env": {
            "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE": "true"
          }
        }
      }
    ],

    "kerberos_principal": {
      "principal_name": "user1/_HOST@EXAMPLE.COM",
      "keytab": "file:///etc/security/keytabs/user1.keytab"
    }
}

If I leave out the "kerberos_principal" section completely, I receive this error at service submission:

{"diagnostics":"Kerberos principal or keytab is missing."}

If I use a principal without the "_HOST" portion, I receive this error at service submission:

{"diagnostics":"Kerberos principal (user1@EXAMPLE.COM) does  not contain a hostname."}

If the keytab does not exist on the worker node, I receive this error in the application log:

org.apache.hadoop.service.ServiceStateException: java.io.IOException: 
SASL is configured for registry, but neither keytab/principal nor 
java.security.auth.login.config system property are specified

Re: Dockerized YARN services with Kerberos

gsaha — Sat, 10 Nov 2018 08:51:16 GMT

You can upload the keytab from any one host to hdfs and then set "keytab" value to that path, something like "hdfs:///user/user1/user1.keytab". Note, the principal_name in that case cannot contain _HOST anymore and has to be expanded to the hostname from where you chose to upload the keytab, so something like "user1/host1.example.com@EXAMPLE.COM".

Re: Dockerized YARN services with Kerberos

shjelmfelt — Sat, 10 Nov 2018 09:08:41 GMT

That worked. I just uploaded one of the keytabs into hdfs:/user/user1/user1_host1.keytab and updated the "kerberos_principal" section as follows. Is there a plan to remove the hostname requirement?

Thanks, @Gour Saha!

"kerberos_principal": {
  "principal_name": "user1/host1.example.com@EXAMPLE.COM",
  "keytab": "hdfs:/user/user1/user1_host1.keytab"
}

Re: Dockerized YARN services with Kerberos

gsaha — Sat, 10 Nov 2018 09:14:02 GMT

Do you mean will we support principal of the format "user@EXAMPLE.COM"?

Re: Dockerized YARN services with Kerberos

shjelmfelt — Sat, 10 Nov 2018 09:18:39 GMT

Yes, that is what I meant.

Re: Dockerized YARN services with Kerberos

gsaha — Sat, 10 Nov 2018 11:16:17 GMT

That's because this keytab is used by YARN Service master which needs a service principal and not a user principal. Its all towards thwarting replay attacks.

Re: Dockerized YARN services with Kerberos

billie — Mon, 12 Nov 2018 23:53:07 GMT

YARN-7787 is open to discuss the issue, but there is no clear solution.