KMS gets 500 error when decrypting files when being accessed from another one way trust realm.

[#1]

SYMPTOM: KMS gets 500 error when decrypting files when being accessed from another one way trust realm.

ERRORS:

Command line error:

[root@support ~]$ hdfs dfs -cat /zone_encr3/abc1.txt 
cat: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 500, message: Internal Server Error

KMS Stack Trace error:

2016-05-04 15:44:21,677 ERROR [webservices-driver] - Servlet.service() for servlet [webservices-driver] in context with path [/kms] threw exception
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to [email protected]
   at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
   at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377)
   at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:422)
   at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347)
   at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
   at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
   at org.apache.hadoop.crypto.key.kms.server.KMSAuthenticationFilter.doFilter(KMSAuthenticationFilter.java:129)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
   at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
   at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
   at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.lang.Thread.run(Thread.java:745)

ROOT CAUSE: There is a one way trust from another realm and auth to local rules are not given to the KMS configuration.

RESOLUTION: hadoop.kms.authentication.kerberos.name.rules property needs to have the auth to local rules.

By default this property is set to DEFAULT. If you replace this value with the value from auth_to_local (core-site.xml) and restart ranger KMS service then we will see a user from another realm is able to decrypt the file successfully.

Copy the rules from core-site.xml (auth_to_local property) to "hadoop.kms.authentication.kerberos.name.rules" in KMS and restart KMS service.

Note:- When you paste the rules from auth_to_local to "hadoop.kms.authentication.kerberos.name.rules", the rules are pasted with space separated values instead of newline. This is fine.