How can I create Kerberos principals during cluster creation?

[#1]

Using Ambari 2.2.2 to install HDP 2.4.2, I want to use a blueprint and cluster template to create a new kerberized cluster. I have tried a number of different approaches (submit a kerberos descriptor json seperately and invoke it with kerberos_descriptor_reference in blueprint; embed kerberos_descriptor json in blueprint; embed kerberos_descriptor json in cluster template) but the principals in my kerberos descriptor are not being created, and stack/service/component defaults are used instead. I assume that I've had typos or logical errors in my attempts.

Since I am deploying with OneFS, I need to modify a handful of principals per the recent guide linked from this question. The most direct mention of principals I could find was in AMBARI-14516 by @Robert Levas. So I tried to mimic it exactly in one of my attempts. I did not include a kerberos_descriptor field in my Blueprint, just used this:

  "Blueprints" : {
    "stack_name" : "HDP",
    "stack_version" : "2.4",
    "security" : {
      "type" : "KERBEROS"
    }
  }

Then in my cluster template I included this:

  "security" : {
    "type" : "KERBEROS",
      "kerberos_descriptor": {
        "properties": {
          "realm" : "WORKINGREALM.COM",
          "keytab_dir" : "/etc/security/keytabs"
        },
      "identities" : [
        {
          "principal" : {
            "type" : "user",
            "value" : "ambari-qa@WORKINGREALM.COM"
          },
          "name" : "smokeuser"
        },
        {
          "principal" : {
            "type" : "user",
            "value" : "hdfs@WORKINGREALM.COM"
          },
          "name" : "hdfs"
        }
      ]
    }
  },
  "configurations" : [
    {
      "kerberos-env": {
        "properties_attributes" : { },
        "properties" : {
          "realm" : "WORKINGREALM.COM",
          "kdc_type" : "mit-kdc",
          "kdc_host" : "therealhost.fqdn.com",
          "admin_server_host" : "therealhost.fqdn.com"
        }
      }
    },
    {
      "krb5-conf": {
        "properties_attributes" : { },
        "properties" : {
          "domains" : "WORKINGREALM.COM",
          "manage_krb5_conf" : "true"
        }
      }
    }
  ]

The ambari-qa upn is created properly per my definition here, but the hdfs upn still uses the default structure of hdfs-${cluster_name}@${realm}. How do I override that?

I tried nesting the hdfs upn in the "services" : [ { "components" : {} } ] structure, but didn't have success with that either.

If you can tell me a way that definitely does work with Ambari 2.2.2, then I can figure out what I was doing wrong when I made that type of attempt.

Another question to consider: "How do I debug why a Kerberos Descriptor json block is not having my anticipated result?" Because there is nothing in the ambari-server logs whether it succeeds or fails. Simply "Processing principal, xyz" and "Creating keytab file for xyx" with no indication of what led to that.

(By the way -- and I'm getting off track here -- the Automated+Kerberization guide is fantastic. I am casting my vote for quick update to the Kerberos Descriptor section to include user submitted KDs, blueprints with internal KD, and cluster template with KD, and how those overlay with stack/service/component KDs. Here are some of the questions that I have that go unanswered because only stack/service/component is discussed... Can my cluster template json reference values in a kerberos descriptor json and in a stack json, or do they need to be declared at that level? Which overrides which? Why is there an escaped text blob in the KD json but an artifact blob in the KD json extracted from a blueprint?)

[#2]

@Tim Veil has a helpful example to start with:

https://community.hortonworks.com/questions/38357/enable-kerberos-via-ambari-blueprint.html

[#3]

I guess I wasn't clear enough in the title. It should be "How do I create custom Kerberos principals during cluster creation?"

As I said in the body, I'm able to Kerberize just fine, and Ambari is successfully creating principals in my KDC. However, I need to customize a number of principals. I've only been successful with smokeuser aka ambari-qa.

Specifically I need to change:

- UPNs: HDFS, Accumulo Tracer, Accumulo, HBase, Storm, Spark

- SPNs: HDFS NameNode, Yarn Resource Manager, Yarn Node Manager, Mapred2 Job History Server

If I could get an example of one non-smokeuser UPN and one SPN, and assurance that it works in both a blueprint and a cluster template, I think I can do the rest.

[#4 · Accepted solution]

@Rob Ketcherside, You should be able to do what you are currently doing, but you will also need to add kerberos descriptor entries for the custom service. See https://cwiki.apache.org/confluence/display/AMBARI/Automated+Kerberizaton#AutomatedKerberizaton-Desc... for information on creating this entry.

As as quick start example, you will want to add a block under the Kerberos descriptor services section (in the blueprint) for your service...

{
  ...,
  "services" : {
    ...,
    {
      "name" : "OneFS",
      "identities": [ 
        ...
      ],
      "components": [ 
        ...
      ],
      ...
    },
    ...
  },
  ...
}

[#5]

@Robert Levas Fantastic, thanks for your help! That small sample was the reassurance I needed on what path to follow.

A bit of background: OneFS actually isn't a separate service in Ambari. The OneFS cluster reveals itself as a single FQDN, so that is simply added as an additional host within Ambari Server. For clarity, here's what the blueprint block looks like for OneFS:

  "host_groups" : [
    {
      "name" : "onefs_group",
      "components" : [
        {"name" : "DATANODE"},
        {"name" : "NAMENODE"},
        {"name" : "SECONDARY_NAMENODE"},
        {"name" : "KERBEROS_CLIENT"}
      ],
      "cardinality" : "1"
    },

I'll update the original question with the security block I used.

Robert, this post you did in April was also really helpful as I tracked down my last typo. https://community.hortonworks.com/articles/28734/obtaining-a-listing-of-expected-kerberos-identitie....

Thanks again!

[#6]

Here's what I used after guidance from @Robert Levas

  "security" : {
    "type" : "KERBEROS",
    "kerberos_descriptor": {
      "properties": {
        "realm" : "YOURREALMHERE",
        "keytab_dir" : "/etc/security/keytabs"
      },
      "identities" : [
        {
          "principal" : {
            "type" : "user",
            "value" : "ambari-qa@${realm}"
          },
          "name" : "smokeuser"
        }
      ],
      "services" : [
        {
          "name" : "HDFS",
          "components" : [
            {
              "name" : "NAMENODE",
              "identities" : [
                {
                  "principal" : {
                    "configuration" : "hadoop-env/hdfs_principal_name",
                    "type" : "user",
                    "local_username" : "hdfs",
                    "value" : "hdfs@${realm}"
                  },
                  "name" : "hdfs"
                }
              ]
            }
          ]
        },
        {
          "name" : "MAPREDUCE2",
          "components" : [
            {
              "name" : "HISTORYSERVER",
              "identities" : [
                {
                  "principal" : {
                    "configuration" : "mapred-site/mapreduce.jobhistory.principal",
                    "type" : "service",
                    "local_username" : "mapred",
                    "value" : "mapred/_HOST@${realm}"
                  },
                  "name" : "history_server_jhs"
                }
              ]
            }
          ]
        },
        {
          "name" : "SPARK",
          "identities" : [
            {
              "principal" : {
                "configuration" : "spark-defaults/spark.history.kerberos.principal",
                "type" : "user",
                "local_username" : "spark",
                "value" : "spark@${realm}"
              },
              "name" : "sparkuser"
            }
          ]
        },
        {
          "name" : "ACCUMULO",
          "identities" : [
            {
              "principal" : {
                "configuration" : "accumulo-env/accumulo_principal_name",
                "type" : "user",
                "local_username" : "accumulo",
                "value" : "accumulo@${realm}"
              },
              "name" : "accumulo"
            },
            {
              "principal" : {
                "configuration" : "accumulo-site/trace.user",
                "type" : "user",
                "local_username" : "accumulo",
                "value" : "accumulo@${realm}"
              },
              "name" : "accumulo_tracer"
            }
          ]
        },
        {
          "name" : "HBASE",
          "identities" : [
            {
              "principal" : {
                "configuration" : "hbase-env/hbase_principal_name",
                "type" : "user",
                "local_username" : "hbase",
                "value" : "hbase@${realm}"
              },
              "name" : "hbase"
            }
          ]
        },
        {
          "name" : "YARN",
          "components" : [
            {
              "name" : "NODEMANAGER",
              "identities" : [
                {
                  "principal" : {
                    "configuration" : "yarn-site/yarn.nodemanager.principal",
                    "type" : "service",
                    "local_username" : "yarn",
                    "value" : "yarn/_HOST@${realm}"
                  },
                  "name" : "nodemanager_nm"
                }
              ]
            },
            {
              "name" : "RESOURCEMANAGER",
              "identities" : [
                {
                  "principal" : {
                    "configuration" : "yarn-site/yarn.resourcemanager.principal",
                    "type" : "service",
                    "local_username" : "yarn",
                    "value" : "yarn/_HOST@${realm}"
                  },
                  "name" : "resource_manager_rm"
                }
              ]
            }
          ]
        },
        {
          "name" : "STORM",
          "identities" : [
            {
              "principal" : {
                "configuration" : "storm-env/storm_principal_name",
                "type" : "user",
                "value" : "storm@${realm}"
              },
              "name" : "storm_components"
            }
          ]
        },
        {
          "name" : "FALCON",
          "configurations" : [
            {
              "falcon-startup.properties" : {
                "*.dfs.namenode.kerberos.principal" : "hdfs/_HOST@${realm}"
              }
            }
          ]
        }
      ]
    }
  },
  "configurations" : [
    {
      "kerberos-env": {
        "properties_attributes" : { },
        "properties" : {
          "realm" : "YOURREALM",
          "kdc_type" : "mit-kdc",
          "kdc_host" : "my-real-kdc.emc.com",
          "admin_server_host" : "my-real-ambari-server.emc.com"
        }
      }
    },
    {
      "krb5-conf": {
        "properties_attributes" : { },
        "properties" : {
          "domains" : "YOURREALM",
          "manage_krb5_conf" : "true"
        }
      }
    }
  ]