Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Kerberos authorization issue (Ranger) - GSSException (Checksum failed)

Labels:
avatar
author-rank epicurean_sagar
New Contributor

Created ‎05-28-2019 01:08 AM

Hi,


I am facing an issue with the SPNEGO authentication w.r.t Ranger. The Ranger HDFS plugin fails to obtain policies because of the same. The Ranger Admin is on a kerberized node (standalone) and the HDFS plugin is setup in the namenode of my Hadoop cluster.


Here is the response I get for the curl request to obtain Ranger policy 

* TCP_NODELAY set
* Connected to <RANGER_ADMIN_HOST_FQDN> (<RANGER_ADMIN_HOST_IP>) port 6080 (#0)
> GET /service/plugins/secure/policies/download/hadoopdev?lastKnownVersion=-1&lastActivationTime=0&pluginId=hdfs@localhost-hadoopdev&clusterName HTTP/1.1
> Host: <RANGER_ADMIN_HOST_FQDN>:6080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< Set-Cookie: RANGERADMINSESSIONID=64653490F3D7C23B0253FB7F3C92C60C; Path=/; HttpOnly
< WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; Domain=; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Length: 0
< Date: Mon, 27 May 2019 09:30:26 GMT
<
* Connection #0 to host <RANGER_ADMIN_HOST_FQDN> left intact
* Issue another request to this URL: 'http://<RANGER_ADMIN_HOST_FQDN>:6080/service/plugins/secure/policies/download/hadoopdev?lastKnownVersion=-1&lastActivationTime=0&pluginId=hdfs@localhost-hadoopdev&clusterName'
* Found bundle for host <RANGER_ADMIN_HOST_FQDN>: 0x2086c80 [can pipeline]
* Re-using existing connection! (#0) with host <RANGER_ADMIN_HOST_FQDN>
* Connected to <RANGER_ADMIN_HOST_FQDN> (<RANGER_ADMIN_HOST_IP>) port 6080 (#0)
* Server auth using Negotiate with user ''
> GET /service/plugins/secure/policies/download/hadoopdev?lastKnownVersion=-1&lastActivationTime=0&pluginId=hdfs@localhost-hadoopdev&clusterName HTTP/1.1
> Host: <RANGER_ADMIN_HOST_FQDN>:6080
> Authorization: Negotiate YIIC4gYGKwYBBQUCoIIC1jCCAtKgDTALBgkqhkiG9xIBAgKiggK/BIICu2CCArcGCSqGSIb3EgECAgEAboICpjCCAqKgAwIBBaEDAgEOogcDBQAgAAAAo4IBm2GCAZcwggGToAMCAQWhDhsMRUMyLklOVEVSTkFMojAwLqADAgEDoScwJRsESFRUUBsdaXAtMTAtMTA3LTUwLTE4My5lYzIuaW50ZXJuYWyjggFIMIIBRKADAgESoQMCAQGiggE2BIIBMoUtDfa1ba9b/LeZi3u8LERW1dx3Y/xsKr8USRqV8idJ2gD/UzTaGmBEcx2KGot4DuQaHaDOh4lQrM62+ky37fxQjFEM2nO6sR92FFmJgF3hKXuK5C/rdTTgaV7cuhkN0CiYeOre03HOsevjJQ99IZdk25IRadypPrSOYYKqM68A4yFqRcwgPai4dgoNILaVqeJrf3djaKKGQn9wXmwBDJMmynqv2UBx2gofEewq0gTlNoBJcyTRFl9n0pDughllI0EDj3nMMJRPY2v6vqxBYsi6MKG1qY7czc8XNbJuXshxisu0/6HbnYEf9hgNcvKAfgCVYewymMZbJy7Vm938/TjEAlQGbTJAkQl1683skTp6PeQ6NMFKOUMYSOb6W7PXPDpRPDtNyytJhxyPaBloWbEbOaSB7TCB6qADAgESooHiBIHfspT7kQlyEE7htuwcA9tzJDM0cnAhghoT5pK/xpBNxGFLu6svjGI4pJoVguCGR690anI6YDCXBK0RUyNgrN8W8ShBwuoOUD01XSHm9uoZu85/OIg9b6qgVM7u+NbPRz5r971OdU8+2CswXdJ/1cwc+NCHaF9752oartLen0kKaLHYqFUzlxLFnfJO5oo6PpGGqaopYyFvm/KJg/FZY4mkWVOgNOl+ziXXO7JXKrvt/3YEKd1006t7Yenp4bmUbhZwI5D321hap085hvwcqPaYdk3BBIG8cgyNDzyZQ3hnSQ==
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Server: Apache-Coyote/1.1
< Set-Cookie: RANGERADMINSESSIONID=DC3BB3EF1F34ACA6759F6731954B2C88; Path=/; HttpOnly
< Set-Cookie: hadoop.auth=; Path=/; Domain=; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 1056
< Date: Mon, 27 May 2019 09:30:26 GMT
<
<!doctype html><html lang="en"><head><title>HTTP Status 403 – Forbidden</title><style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}</style></head><body><h1>HTTP Status 403 – Forbidden</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</p><p><b>Description</b> The server understood the request but refuses to authorize it.</p><hr class="line" /><h3>Apache * Closing connection


The request returns HTTP 403 with error GSSException:...(Mechanism Level: Checksum Failed). Please find below the stack trace of the same from the Ranger admin logs.

Found KeyTab /etc/security/keytabs/HTTP.keytab for HTTP/<HOST_FQDN>@REALM
Found KeyTab /etc/security/keytabs/HTTP.keytab for HTTP/<HOST_FQDN>@REALM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/<HOST_FQDN>@REALM
Added key: 17version: 2
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:398)
        at org.apache.ranger.security.web.filter.RangerKrbFilter.doFilter(RangerKrbFilter.java:447)
        at org.apache.ranger.security.web.filter.RangerKRBAuthenticationFilter.doFilter(RangerKRBAuthenticationFilter.java:385)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.apache.ranger.security.web.filter.RangerSSOAuthenticationFilter.doFilter(RangerSSOAuthenticationFilter.java:234)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:906)
        at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:556)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:365)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347)
        ... 45 more
Caused by: KrbException: Checksum failed
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281)
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829)
        ... 56 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
        at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
        ... 62 more


Here are the details pertaining to the kerberos configuration and SPN(s) in both the nodes. I am wondering if the difference in the enctype is the root cause of the issue. Could I be missing any SPN? I am quite new to Kerberos/SPNEGO.


-> enctypes properties from krb5.conf from <RANGER_ADMIN_HOST> - same as that in the Hadoop namenode

default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1
default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sha1
permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96 des3-cbc-sh


-> Kerberos/SPN info from the Hadoop namenode

[root@HADOOP_NAMENODE_HOST hadoop]# kdb5_util list_mkeys
Master keys for Principal: K/M@REALM
KVNO: 1, Enctype: des3-cbc-sha1, Active on: Thu Jan 01 00:00:00 UTC 1970 *
[root@HADOOP_NAMENODE_HOST hadoop]# kadmin.local
Authenticating as principal root/admin@REALM with password.
kadmin.local:  getprinc HTTP/HADOOP_NAMENODE_HOST_FQDN@REALM
Principal: HTTP/HADOOP_NAMENODE_HOST_FQDN@REALM
Expiration date: [never]
Last password change: Sun May 19 13:24:29 UTC 2019
Password expiration date: [never]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Sun May 19 13:24:29 UTC 2019 (kadmin/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 3
Key: vno 2, aes256-cts-hmac-sha1-96
Key: vno 2, aes128-cts-hmac-sha1-96
Key: vno 2, des3-cbc-sha1
MKey: vno 1
Attributes:
Policy: [none]
kadmin.local:  getprinc HTTP/RANGER_ADMIN_HOST_FQDN@REALM
Principal: HTTP/RANGER_ADMIN_HOST_FQDN@REALM
Expiration date: [never]
Last password change: Sun May 26 00:18:43 UTC 2019
Password expiration date: [never]
Maximum ticket life: 0 days 10:00:00
Maximum renewable life: 7 days 00:00:00
Last modified: Sun May 26 00:18:43 UTC 2019 (hdfs/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 3
Key: vno 1, aes256-cts-hmac-sha1-96
Key: vno 1, aes128-cts-hmac-sha1-96
Key: vno 1, des3-cbc-sha1
MKey: vno 1
Attributes:
Policy: [none]


-> Kerberos/SPN info from the Ranger admin server

[root@RANGER_ADMIN_HOST admin]# kdb5_util list_mkeys
Master keys for Principal: K/M@REALM
KVNO: 1, Enctype: aes256-cts-hmac-sha1-96, Active on: Thu Jan 01 00:00:00 UTC 1970 *
[root@RANGER_ADMIN_HOST admin]# kadmin.local
Authenticating as principal HTTP/admin@REALM with password.
kadmin.local:  getprinc HTTP/RANGER_ADMIN_HOST_FQDN@REALM
Principal: HTTP/RANGER_ADMIN_HOST_FQDN@REALM
Expiration date: [never]
Last password change: Sat May 25 01:09:03 UTC 2019
Password expiration date: [never]
Maximum ticket life: 1 day 00:00:00
Maximum renewable life: 0 days 00:00:00
Last modified: Sat May 25 01:09:03 UTC 2019 (root/admin@REALM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 2
Key: vno 2, aes256-cts-hmac-sha1-96
Key: vno 2, aes128-cts-hmac-sha1-96
MKey: vno 1
Attributes:
Policy: [none]


I would greatly appreciate any help. Thank you.

13,516 Views
0 Kudos
2 REPLIES 2

avatar
author-rank VR46 author-rank
Guru

Created ‎05-29-2019 03:09 AM

Hello @Vidya Sagar S,

I think I see your problem. It's with the key of HTTP/RANGER_FQDN principal. Most probably, the key of Ranger SPNEGO principal (HTTP/RANGER_FQDN) has been changed/updated in Kerberos database but not in the spnego.service.keytab on Ranger node. Hence "checksum failed" error.

To confirm this, please get the output of these commands on Ranger host:

# klist -kt /etc/security/keytabs/spnego.service.keytab
# kinit <any-working-principal>
# kvno HTTP/<RANGER_FQDN>

If the Key Version Number (kvno) in the first command output and last command output doesn't match, then that's the issue.

To fix this, get the key of HTTP/RANGER_FQDN out in a new keytab and replace spnego.service.keytab on Ranger host. Restart Ranger admin from Ambari.

Hope this helps!

13,322 Views
0 Kudos

avatar
author-rank epicurean_sagar
New Contributor

Created ‎06-03-2019 05:03 PM

Hello @Vipin Rathor,

Please find below the output of the commands 

[ec2-user@RANGER_HOST ~]$ klist -kt /etc/security/keytabs/HTTP.keytab
Keytab name: FILE:/etc/security/keytabs/HTTP.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 05/27/2019 22:20:26 HTTP/RANGER_FQDN@REALM
   2 05/27/2019 22:20:26 HTTP/RANGER_FQDN@REALM
   2 05/27/2019 22:20:26 HTTP/RANGER_FQDN@REALM
   2 06/01/2019 21:47:42 HTTP/HADOOP_NAMENODE_FQDN@REALM
   2 06/01/2019 21:47:42 HTTP/HADOOP_NAMENODE_FQDN@REALM
   2 06/01/2019 21:47:42 HTTP/HADOOP_NAMENODE_FQDN@REALM
[ec2-user@RANGER_HOST ~]$ kinit -kt /etc/security/keytabs/hdfs.keytab hdfs/RANGER_FQDN@REALM
[ec2-user@RANGER_HOST ~]$ kvno HTTP/RANGER_FQDN@REALM
HTTP/RANGER_FQDN@REALM: kvno = 2

The kvno is the same. In the second step, I just did kinit on a keytab/principal I had created.

13,322 Views
0 Kudos
Announcements
What's New @ Cloudera
[RELEASED] Cloudera Streaming Analytics 1.14 for Cloudera Pu...
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
View More Announcements