Support Questions

rakesh_satheesa · ‎09-14-2018

We have a kerberised hdp cluster ( 2.6.5 ) deployed in AWS. AWS network architecture is in such a way that all the hdp component nodes are under private subnet and the access to them is only via ssh from bastion node which is in public subnet. We have enabled all the web components ( Storm UI, Metron UI, Metron Management UI etc ) available outside via AWS ELB load balancer to the outside world.

Our kerberos server and kdc admin is available outside via ssh tunneling via bastion node, This is for the external accessing client to authenticate eg : Spengo.

When we access our storm UI via browser with proper step taken for to pass spengo authentication, We are getting 403 error even with proper keytab and principal.

Error getting in /var/log/storm/ui.out in storm UI hosted node

Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/sdssystemmaster2@EXAMPLE.COM
Looking for keys for: HTTP/sdssystemmaster2@EXAMPLE.COM
Found unsupported keytype (3) for HTTP/sdssystemmaster2@EXAMPLE.COM
MemoryCache: add 1536315369/301662/8ABC886166F6808EA668D561462EDD37/metron@EXAMPLE.COM to metron@HOST.

Steps followed :

1- Installed kerberos client

2- Copied krb5.conf file from kerberose node to local file krb5.ini and configured

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = EXAMPLE.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  EXAMPLE.COM = {
    admin_server = localhost
    kdc = localhost
  }

3- Copied keytab file of principal - HTTP/sdssystemmaster2@EXAMPLE.COM

spnego.service.keytab

3- kinit executed () and ticket seems to be generated fine (screenshot added)

4- Configured firefox about:config

network.negotiate-auth.trusted-uris : loadbalancer-url 
network.negotiate-auth.delegation-uris : loadbalancer-url 
network.negotiate-auth.gsslib : C:\Program Files\MIT\Kerberos\bin\gssapi64.dll 
network.negotiate-auth.using-native-gsslib : false

5- Loaded the storm UI

Storm UI spengo Kerberos configuration

ui.filter : org.apache.hadoop.security.authentication.server.AuthenticationFilter

ui.filter.params : {'type': 'kerberos', 'kerberos.principal': '{{storm_ui_jaas_principal}}', 'kerberos.keytab':'{{storm_ui_keytab_path}}' , 'kerberos.name.rules': 'DEFAULT'}


storm_ui_keytab : /etc/security/keytabs/spnego.service.keytab
storm_ui_principal_name : HTTP/_HOST@EXAMPLE.COM

rlevas · ‎09-14-2018

Key type 3 is DES_CBC_MD5, which is pretty much deprecated (see https://www.opencore.com/blog/2017/3/kerberos-encryption-types/), but by default Ambari requests/creates keytab entries using this type for backwards compatibility. Your KDC is probably rejecting keys encrypted with this type.

To fix this, you should go into the Kerberos service settings and edit the "Encryption Type" value under the "Advanced kerberos-env" section. The default value is "aes des3-cbc-sha1 rc4 des-cbc-md5". Change it to "aes des3-cbc-sha1 rc4".

You will also want to update the "krb5-conf template" value under "Advanced krb5-conf" to add the following under the "[libdefaults]" section:

allow_weak_crypto = false

After saving the changes and restarting the Kerberos service (which ensure the krb5.conf file is synced up), you should restart all of the services. If you still see issues, maybe regenerate all keytab files (Admin->Kerberos) and then restart all services. However depending on the KDC implementation you may or may not see a change in the generated keytab files. By default they will look like

[root@c7401 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

View solution in original post

rlevas · ‎09-28-2018

@Rakesh S

In a load balancer scenario, the browser should ask for a Kerberos ticket to access the web server on the load balancer. The load balancer should forward the request on to the real service. The real service should be able to validate the Kerberos token using the load balancer's principal and keytab entry. So the load balancer's keytab entry needs to be in the service's configured keytab file.

However for all this to work, the service needs to understand that a load balancer is in the middle and behave properly. So there are some details here, related to Storm that some one familiar with storm needs to answer. For example does Storm support this scenario?

I seem to have lost track of the issue here... the subject of this tread is related to an unsupported key. I think we solved this and have now moved on to another issue?

rakesh_satheesa · ‎10-03-2018

@Robert Levas

Thanks for your support

irshadkt_mec · ‎10-03-2018

@Robert Levas : Thanks a lot for the solution, 'unsupported type' issue has been resolved after kerberose config changes. I have raised another issue for storm spenago load balancer related here.

Cloudera Community

Support Questions

403 error with 'unsupported key type' message upon accessing spengo enabled storm UI.