Created 10-01-2015 11:48 PM
All users are defined locally without any AD integration. The cluster is Kerberized on HDP 2.3 and Ambari 2.1.0. The knit fails as follow:
kinit: krb5_init_creds_set_keytab: Failed to find kafka/hdpblv10.t-mobile.lab@T-MOBILE.LAB in keytab FILE:/etc/security/keytabs/kafka.service.keytab (unknown enctype) .
The Storm UI uses SPNEGO AUTH when in Kerberos mode. Before accessing the UI, you have to we configured our browser for SPNEGO authorization before accessing the UI as follow:
Safari: no changes needed. Firefox: 1) Go to about:config and search for network.negotiate-auth.trusted-uris. 2) Double-click and add the following value: "http://storm-ui-hostname:ui-port" 3) Replace the storm-ui-hostname with the hostname where your UI is running. 4) Replace the ui-port with the Storm UI port. Google-chrome: from the command line, issue: google-chrome --auth-server-whitelist="storm-ui-hostname" --auth-negotiate-delegate-whitelist="storm-ui-hostname" Internet Explorer: 1) Configure trusted websites to include "storm-ui-hostname". 2) Allow negotiation for the UI website.
Created 10-01-2015 11:52 PM
ui.filter.params:"type": "kerberos""kerberos.principal": "HTTP/nimbus.witzend.com""kerberos.keytab": "/vagrant/keytabs/http.keytab""kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
Note that rules are listed as string without commas.
2. You will need to create mapping for MIT Domain KDC and correlated resource used for the Domain, in this case Storm UI. You will need to execute following commands on Windows workstation from the command line:
ksetup /AddKDC $DOMAIN $KDC
ksetup /AddHostToRealmMap $hadoop_resource $Domain
Note that this adds registry entries in:
HKLM\System\CurrentControlSet\Control\Lsa\Kerberos\HostToRealm
If you need to troubleshoot the issues you can try accessing Storm UI within the cluster using curl command. For example:
curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt http://storm-ui-hostname:8080/api/v1/cluster/summary
This will be helpful to see if kerberos UI configs are working.
In order to isolate the issue you can use storm service keytabs as well as user principles.
Another important thing to check is to make sure that trust is working properly and encryption types match on both KDCs.
Created 10-01-2015 11:52 PM
ui.filter.params:"type": "kerberos""kerberos.principal": "HTTP/nimbus.witzend.com""kerberos.keytab": "/vagrant/keytabs/http.keytab""kerberos.name.rules": "RULE:[2:$1@$0]([jt]t@.*EXAMPLE.COM)s/.*/$MAPRED_USER/ RULE:[2:$1@$0]([nd]n@.*EXAMPLE.COM)s/.*/$HDFS_USER/DEFAULT"
Note that rules are listed as string without commas.
2. You will need to create mapping for MIT Domain KDC and correlated resource used for the Domain, in this case Storm UI. You will need to execute following commands on Windows workstation from the command line:
ksetup /AddKDC $DOMAIN $KDC
ksetup /AddHostToRealmMap $hadoop_resource $Domain
Note that this adds registry entries in:
HKLM\System\CurrentControlSet\Control\Lsa\Kerberos\HostToRealm
If you need to troubleshoot the issues you can try accessing Storm UI within the cluster using curl command. For example:
curl -i --negotiate -u:anyUser -b ~/cookiejar.txt -c ~/cookiejar.txt http://storm-ui-hostname:8080/api/v1/cluster/summary
This will be helpful to see if kerberos UI configs are working.
In order to isolate the issue you can use storm service keytabs as well as user principles.
Another important thing to check is to make sure that trust is working properly and encryption types match on both KDCs.
Created 10-02-2015 10:24 AM
Regarding the kinit error. Did you execute this manually on the client host or was the error message retrieved from a log file a node in the cluster?
If you received this message while kinit-ing manually on your client host, why are you attempting to authenticate using a service principal, rather than your user principal?
Can you list the contents of the keytab file?
klist -kte /etc/security/keytabs/kafka.service.keytab
Does it contain entries different than the hdfs keytab file?
klist -kte /etc/security/keytabs/hdfs.headless.keytab
Both listings should have the same number of keytab entries with the same encryption types. For example:
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab KVNO Timestamp Principal ---- ----------------- ---------------------------------------------- 1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (arcfour-hmac) 1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (des3-cbc-sha1) 1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (des-cbc-md5) 1 10/02/15 10:19:29 hdfs-c1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
Created 10-02-2015 11:56 AM
These were the steps I used to get the Storm UI working on my Mac on kerborized HDP 2.3 (search for 'Open kerborized browser'):