403 error with 'unsupported key type' message upon accessing spengo enabled storm UI.

[#1]

We have a kerberised hdp cluster ( 2.6.5 ) deployed in AWS. AWS network architecture is in such a way that all the hdp component nodes are under private subnet and the access to them is only via ssh from bastion node which is in public subnet. We have enabled all the web components ( Storm UI, Metron UI, Metron Management UI etc ) available outside via AWS ELB load balancer to the outside world.

Our kerberos server and kdc admin is available outside via ssh tunneling via bastion node, This is for the external accessing client to authenticate eg : Spengo.

When we access our storm UI via browser with proper step taken for to pass spengo authentication, We are getting 403 error even with proper keytab and principal.

Error getting in /var/log/storm/ui.out in storm UI hosted node

Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/sdssystemmaster2@EXAMPLE.COM
Looking for keys for: HTTP/sdssystemmaster2@EXAMPLE.COM
Found unsupported keytype (3) for HTTP/sdssystemmaster2@EXAMPLE.COM
MemoryCache: add 1536315369/301662/8ABC886166F6808EA668D561462EDD37/metron@EXAMPLE.COM to metron@HOST.

Steps followed :

1- Installed kerberos client

2- Copied krb5.conf file from kerberose node to local file krb5.ini and configured

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = EXAMPLE.COM
  ticket_lifetime = 24h
  dns_lookup_realm = false
  dns_lookup_kdc = false
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  EXAMPLE.COM = {
    admin_server = localhost
    kdc = localhost
  }

3- Copied keytab file of principal - HTTP/sdssystemmaster2@EXAMPLE.COM

spnego.service.keytab

3- kinit executed () and ticket seems to be generated fine (screenshot added)

4- Configured firefox about:config

network.negotiate-auth.trusted-uris : loadbalancer-url 
network.negotiate-auth.delegation-uris : loadbalancer-url 
network.negotiate-auth.gsslib : C:\Program Files\MIT\Kerberos\bin\gssapi64.dll 
network.negotiate-auth.using-native-gsslib : false

5- Loaded the storm UI

Storm UI spengo Kerberos configuration

ui.filter : org.apache.hadoop.security.authentication.server.AuthenticationFilter

ui.filter.params : {'type': 'kerberos', 'kerberos.principal': '{{storm_ui_jaas_principal}}', 'kerberos.keytab':'{{storm_ui_keytab_path}}' , 'kerberos.name.rules': 'DEFAULT'}


storm_ui_keytab : /etc/security/keytabs/spnego.service.keytab
storm_ui_principal_name : HTTP/_HOST@EXAMPLE.COM

[#2 · Accepted solution]

Key type 3 is DES_CBC_MD5, which is pretty much deprecated (see https://www.opencore.com/blog/2017/3/kerberos-encryption-types/), but by default Ambari requests/creates keytab entries using this type for backwards compatibility. Your KDC is probably rejecting keys encrypted with this type.

To fix this, you should go into the Kerberos service settings and edit the "Encryption Type" value under the "Advanced kerberos-env" section. The default value is "aes des3-cbc-sha1 rc4 des-cbc-md5". Change it to "aes des3-cbc-sha1 rc4".

You will also want to update the "krb5-conf template" value under "Advanced krb5-conf" to add the following under the "[libdefaults]" section:

allow_weak_crypto = false

After saving the changes and restarting the Kerberos service (which ensure the krb5.conf file is synced up), you should restart all of the services. If you still see issues, maybe regenerate all keytab files (Admin->Kerberos) and then restart all services. However depending on the KDC implementation you may or may not see a change in the generated keytab files. By default they will look like

[root@c7401 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   2 09/14/2018 15:06:22 HTTP/c7401.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

[#3]

Thanks

Unsupported key type issue is solved.

Now getting another error in /var/log/krb5kdc.log while accessing storm web ui.

Sep1708:37:50 ip-10-0-2-8.ec2.internal krb5kdc[1536](info): TGS_REQ (6 etypes {181716232526})127.0.0.1: LOOKING_UP_SERVER: authtime 0,  HTTP/sdssystemmaster2@EXAMPLE.COM for HTTP/ec2-18-114-9-244.compute-1.amazonaws.com@EXAMPLE.COM,Servernot found inKerberos database
Sep1708:37:50 ip-10-0-2-8.ec2.internal krb5kdc[1536](info): closing down fd 13
Sep1708:37:50 ip-10-0-2-8.ec2.internal krb5kdc[1536](info): TGS_REQ (6 etypes {181716232526})127.0.0.1: UNKNOWN_SERVER: authtime 0,  HTTP/sdssystemmaster2@EXAMPLE.COM for krbtgt/COMPUTE-1.AMAZONAWS.COM@EXAMPLE.COM,Servernot found inKerberos database
Sep1708:37:50 ip-10-0-2-8.ec2.internal krb5kdc[1536](info): closing down fd 13

[#4]

HTTP/sdssystemmaster2@EXAMPLE.COM is not a valid service principal. The "sdssystemmaster2" part needs to be a FQDN. Like HTTP/host1.example.com@EXAMPLE.COM. Where did that value come from? It is possible that you have a DNS or reverse DNS issue.

[#5]

@Robert Levas

That principal name is masked.

Actually it is in FQDN format (HTTP/sdssystemmaster2.example.com@EXAMPLE.COM)

[#6]

oops.. sorry about that. It is not always clear when hostnames are masked and I have seen issues where the hostname was incorrect (for various reasons).

[#7]

Looking at the error message again, it appears that the issue is with the principal name HTTP/ec2-18-114-9-244.compute-1.amazonaws.com@EXAMPLE.COM. My reason is the same as before... a DNS or reverse DNS issue. Take a look at the KDC to see if this principal exists. If you are using an MIT KDC and have access to the host where the KDC is installed, you can find all of the SPNEGO principals by doing:

kadmin.local -q list_principals | grep HTTP

Ideally they all look similar and the hostname for each of the HTTP/<hostname> principals is an expected hostname.

Keep in mind that both DNS and reverse DNS need to use the same IP address to DNS name mapping, else issues like this will occur.

[#8]

@Robert Levas

List of SPNEGO principals

HTTP/sdssystemedge.example.com@EXAMPLE.COM
HTTP/sdssystemhdf1.example.com@EXAMPLE.COM
HTTP/sdssystemhdf2.example.com@EXAMPLE.COM
HTTP/sdssystemhdf3.example.com@EXAMPLE.COM
HTTP/sdssystemmaster1.example.com@EXAMPLE.COM
HTTP/sdssystemmaster2.example.com@EXAMPLE.COM
HTTP/sdssystemworker1.example.com@EXAMPLE.COM
HTTP/sdssystemworker2.example.com@EXAMPLE.COM
HTTP/sdssystemworker3.example.com@EXAMPLE.COM
HTTP/sdssystemworker4.example.com@EXAMPLE.COM

Turned off Reverse-DNS by setting rdns=false in krb5.conf file.

But still getting same error

HTTP ERROR: 403
Problem accessing /. Reason:
org.apache.hadoop.security.authentication.client.AuthenticationException

---

Powered by Jetty://

[#9]

Turning off rdns in the krb5.conf wont help. I am not even sure the property applies to this type of scenario. The Hadoop libraries do the reverse DNS lookup on their own.

[#10]

@Robert Levas

Could this be a problem related to principal mismatch in client and spengo configuration ?

When I checked `klist` I could see principal name has HTTP/$loadBalancer@EXAMPLE.COM. Where I actually configured HTTP/_HOST@EXAMPLE.COM in ui.filter.params. Is the browser override the principal name as the host ? Can we bypass that ?

My Spnego configuration object -

{'type': 'kerberos', 'kerberos.principal': 'HTTP/_HOST@EXAMPLE.COM', 'kerberos.keytab':'/etc/security/keytabs/spnego.service.keytab' , 'kerberos.name.rules': 'DEFAULT'} 

klist output -

klist Ticket cache: FILE:/tmp/krb5cc_1000

Default principal: HTTP/sdssystemmaster2.example.com@EXAMPLE.COM

Valid starting       Expires              Service principal                   
2018-09-28T10:12:09  2018-09-29T10:12:09  krbtgt/example.com@EXAMPLE.COM     
2018-09-28T10:22:44  2018-09-29T10:12:09  HTTP/elb.amazonaws.com@                                          2018-09-28T10:22:44  2018-09-29T10:12:09  HTTP/elb.amazonaws.com@EXAMPLE.COM 

[#11]

@Rakesh S

In a load balancer scenario, the browser should ask for a Kerberos ticket to access the web server on the load balancer. The load balancer should forward the request on to the real service. The real service should be able to validate the Kerberos token using the load balancer's principal and keytab entry. So the load balancer's keytab entry needs to be in the service's configured keytab file.

However for all this to work, the service needs to understand that a load balancer is in the middle and behave properly. So there are some details here, related to Storm that some one familiar with storm needs to answer. For example does Storm support this scenario?

I seem to have lost track of the issue here... the subject of this tread is related to an unsupported key. I think we solved this and have now moved on to another issue?

[#12]

@Robert Levas

Thanks for your support

[#13]

@Robert Levas : Thanks a lot for the solution, 'unsupported type' issue has been resolved after kerberose config changes. I have raised another issue for storm spenago load balancer related here.