Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error deploying HDFS Ranger Plugin on Kerberised Cluster

avatar
Contributor

Am creating a new cluster through Ambari using blueprints with version 2.4.1.0 of Ambari and HDP 2.5.0, the cluster is running FreeIPA and is kerberised, Ranger deploys fine and no errors are logged in the Ambari logs for the deployment of Ranger Admin or Ranger Usersync services, however when starting the namenode there are errors logged in the startup and the HDFS service is not created in Ranger Web UI.

Have pasted in the relevant logs below and some of the manual commands I have run on the nodes to try to troubleshoot. Any help would be greatly appreciated.

Namenode startup log stderr in Ambari;

2016-11-04 08:45:26,899 - Error in call for getting Ranger service:
 No JSON object could be decoded
2016-11-04 08:54:10,812 - Error in call for creating Ranger service:
 No JSON object could be decoded
2016-11-04 08:54:10,813 - Hdfs Repository creation failed in Ranger admin

Namenode startup log stdout in Ambari;

2016-11-04 08:44:53,766 - checked_call['/usr/bin/kinit -c /var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b -kt /etc/security/keytabs/nn.service.keytab nn/-nn-001.project1@PROJECT1 > /dev/null'] {'user': 'hdfs'}
2016-11-04 08:44:53,855 - checked_call returned (0, '')
2016-11-04 08:44:53,856 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -L -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/710d18ea-f3ae-44d0-804f-b7111ab429e6 -c /var/lib/ambari-agent/tmp/cookies/710d18ea-f3ae-44d0-804f-b7111ab429e6 -w '"'"'%{http_code}'"'"' http://auth-001.project1:6080/login.jsp --connect-timeout 10 --max-time 12 -o /dev/null 1>/tmp/tmppqhoiG 2>/tmp/tmpLIiD5C''] {'quiet': False, 'env': {'KRB5CCNAME': '/var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b'}}
2016-11-04 08:44:53,924 - call returned (0, '')
2016-11-04 08:44:53,925 - call['/usr/bin/klist -s /var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b'] {'user': 'hdfs'}
2016-11-04 08:44:53,980 - call returned (0, '')
2016-11-04 08:44:53,980 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -L -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/3dbe7f89-811d-4dc5-be44-1dac2a6ac2aa -c /var/lib/ambari-agent/tmp/cookies/3dbe7f89-811d-4dc5-be44-1dac2a6ac2aa '"'"'http://auth-001.project1:6080/service/public/v2/api/service?serviceName=PROJECT1_Cluster_hadoop&serviceType=hdfs&isEnabled=true'"'"' --connect-timeout 10 --max-time 12 -X GET 1>/tmp/tmpAMnDmH 2>/tmp/tmp6PLCo5''] {'quiet': False, 'env': {'KRB5CCNAME': '/var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b'}}
2016-11-04 08:44:54,054 - call returned (0, '')
2016-11-04 08:44:54,055 - Will retry 4 time(s), caught exception: Error in call for getting Ranger service:
 No JSON object could be decoded. Sleeping for 8 sec(s)

xa_portal.log from Ranger admin machine auth-001

2016-11-04 08:54:10,828 [http-bio-6080-exec-5] WARN  apache.ranger.security.web.filter.RangerKrbFilter (RangerKrbFilter.java:494) - Authentication exception: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:400)
        at org.apache.ranger.security.web.filter.RangerKrbFilter.doFilter(RangerKrbFilter.java:449)
        at org.apache.ranger.security.web.filter.RangerKRBAuthenticationFilter.doFilter(RangerKRBAuthenticationFilter.java:285)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.apache.ranger.security.web.filter.RangerSSOAuthenticationFilter.doFilter(RangerSSOAuthenticationFilter.java:211)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
        at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:87)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
        at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
        at sun.security.jgss.spnego.SpNegoMechFactory.getCredentialElement(SpNegoMechFactory.java:142)
        at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
        at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
        at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:77)
        at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:160)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:357)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:349)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:349)
        ... 38 more

Manual klist of kerberos keytab cache used by Ambari on nn-001;

/usr/bin/klist /var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b
Ticket cache: FILE:/var/lib/ambari-agent/tmp/curl_krb_cache/ranger_admin_calls_hdfs_cc_7b6e79b8fdca257bc6249b42083c151b
Default principal: nn/nn-001.project1@PROJECT1


Valid starting     Expires            Service principal
04/11/16 08:54:10  05/11/16 08:54:10  krbtgt/PROJECT1@PROJECT1
04/11/16 08:54:10  05/11/16 08:54:10  HTTP/auth-001.project1@PROJECT1

Manual run of curl command used by Ambari to query Ranger services on nn-001;

curl -L -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/3dbe7f89-811d-4dc5-be44-1dac2a6ac2aa -c /var/lib/ambari-agent/tmp/cookies/3dbe7f89-811d-4dc5-be44-1dac2a6ac2aa 'http://auth-001.project1:6080/service/public/v2/api/service?serviceName=PROJECT1_Cluster_hadoop&serviceType=hdfs&isEnabled=true' --connect-timeout 10 --max-time 12 -X GET

<html><head><title>Apache Tomcat/7.0.68 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 403 - GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.68</h3></body></html>

Blueprint is configued to set the xasecure.audit.jaas.Client.option.keyTab to /etc/security/keytabs/rangeradmin.service.keytab and the principal to rangeradmin/_HOST@PROJECT1

klist -kt /etc/security/keytabs/rangeradmin.service.keytab
Keytab name: FILE:/etc/security/keytabs/rangeradmin.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 03/11/16 16:47:02 rangeradmin/auth-001.project1@PROJECT1
   1 03/11/16 16:47:02 rangeradmin/auth-001.project1@PROJECT1
   1 03/11/16 16:47:02 rangeradmin/auth-001.project1@PROJECT1
   1 03/11/16 16:47:02 rangeradmin/auth-001.project1@PROJECT1
1 ACCEPTED SOLUTION

avatar
Contributor

Finally spotted my mistake, was the SPNEGO kerberos configuration in Ambari was incorrect, I had the principal set to HTTP/auth-001@PROJECT1 instead of HTTP/_HOST@PROJECT1.

View solution in original post

1 REPLY 1

avatar
Contributor

Finally spotted my mistake, was the SPNEGO kerberos configuration in Ambari was incorrect, I had the principal set to HTTP/auth-001@PROJECT1 instead of HTTP/_HOST@PROJECT1.