acessing the STORM UI with HDP 2.3 Kerberized cluster.

[#1]

All users are defined locally without any AD integration. The cluster is Kerberized on HDP 2.3 and Ambari 2.1.0. The knit fails as follow:

kinit: krb5_init_creds_set_keytab: Failed to find kafka/hdpblv10.t-mobile.lab@T-MOBILE.LAB in keytab FILE:/etc/security/keytabs/kafka.service.keytab (unknown enctype) .

The Storm UI uses SPNEGO AUTH when in Kerberos mode. Before accessing the UI, you have to we configured our browser for SPNEGO authorization before accessing the UI as follow:

Safari: no changes needed. Firefox: 1) Go to about:config and search for network.negotiate-auth.trusted-uris. 2) Double-click and add the following value: "http://storm-ui-hostname:ui-port" 3) Replace the storm-ui-hostname with the hostname where your UI is running. 4) Replace the ui-port with the Storm UI port. Google-chrome: from the command line, issue: google-chrome --auth-server-whitelist="storm-ui-hostname" --auth-negotiate-delegate-whitelist="storm-ui-hostname" Internet Explorer: 1) Configure trusted websites to include "storm-ui-hostname". 2) Allow negotiation for the UI website.

[#2 · Accepted solution]

1. Make sure ui kerberos auth to local rules are configured properly. Once principle from AD is used for negotiation with MIT KDC, there need to be a rule that translate it to local account in Storm UI node. Many times those can be copied from core-site.xml. For example:

ui.filter.params:"type": "kerberos""kerberos.principal": "HTTP/nimbus.witzend.com""kerberos.keytab": "/vagrant/keytabs/http.keytab""kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"

Note that rules are listed as string without commas.

2. You will need to create mapping for MIT Domain KDC and correlated resource used for the Domain, in this case Storm UI. You will need to execute following commands on Windows workstation from the command line:

ksetup /AddKDC $DOMAIN $KDC

ksetup /AddHostToRealmMap $hadoop_resource $Domain

Note that this adds registry entries in:

HKLM\System\CurrentControlSet\Control\Lsa\Kerberos\HostToRealm

If you need to troubleshoot the issues you can try accessing Storm UI within the cluster using curl command. For example:

curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt http://storm-ui-hostname:8080/api/v1/cluster/summary

This will be helpful to see if kerberos UI configs are working.

In order to isolate the issue you can use storm service keytabs as well as user principles.

Another important thing to check is to make sure that trust is working properly and encryption types match on both KDCs.

[#3]

Regarding the kinit error. Did you execute this manually on the client host or was the error message retrieved from a log file a node in the cluster?

If you received this message while kinit-ing manually on your client host, why are you attempting to authenticate using a service principal, rather than your user principal?

Can you list the contents of the keytab file?

klist -kte /etc/security/keytabs/kafka.service.keytab

Does it contain entries different than the hdfs keytab file?

klist -kte /etc/security/keytabs/hdfs.headless.keytab

Both listings should have the same number of keytab entries with the same encryption types. For example:

Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Timestamp         Principal
---- ----------------- ----------------------------------------------
   1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (arcfour-hmac)
   1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (des3-cbc-sha1)
   1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (des-cbc-md5)
   1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

[#4]

These were the steps I used to get the Storm UI working on my Mac on kerborized HDP 2.3 (search for 'Open kerborized browser'):

https://github.com/abajwa-hw/security-workshops/blob/master/Setup-ranger-23.md#setup-storm-plugin-fo...