question Running into auth_to_local rule issue during Livy kerberos authentication in Support Questions

question Running into auth_to_local rule issue during Livy kerberos authentication in Support Questions https://community.cloudera.com/t5/Support-Questions/Running-into-auth-to-local-rule-issue-during-Livy-kerberos/m-p/296481#M218244 Hello.   We are currently trying to get a proof-of-concept off the ground using Livy as a spark-shell multiplexor in our kerberized CDH 6.1 environment  (I realize Livy is formally supported in CDP 7, but our requirement atm is to get this working under 6 or create our own multiplexor to help reduce wait-time as spark shells start up).  We're running into problems getting Kerberos to properly map names via the auth_to_local rules.    We've completed the following: <OL> <LI>Deployed Livy to a clean, kerberized edge node on the cluster</LI> <LI>Created a service principal for Livy authentication  (see config below)</LI> <LI>Created a user principal for Livy server launch (see config below)</LI> <LI>Generated the keytabs needed to auth both of those principals</LI> <LI>Set the ENV vars needed for Livy to find the active hadoop config / libs  (see below)</LI> </OL> When we launch Livy, we get what seems to be a clean start, and it begins listening on tcp 8998 as expected.  The first time someone tries to hit that through a browser  (using Chrome, and we know it's passing spnego auth to Livy) we get the following:           <LI-CODE lang="markup">20/05/21 13:03:12 WARN ServletHandler: /favicon.ico org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mzeoli@COMPANY.PRI at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745)</LI-CODE>         The complaint about no rules being applied led us to review our auth_to_local rules in core-site.xml  (which is confirmed present in /etc/hadoop/conf), where we see the following local_to_auth rule:           <LI-CODE lang="markup"><name>hadoop.security.auth_to_local</name> <value> RULE:[1:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$// RULE:[2:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$// RULE:[1:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$// RULE:[2:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$// DEFAULT </value></LI-CODE>         If we try to check rule negotiation manually as the livyblauser user (same user that runs livy) from the same edge node, we see the following which looks functional and correct.         <LI-CODE lang="markup">[livyblauser@nydc-pblalivy01 hadoop]$ hadoop org.apache.hadoop.security.HadoopKerberosName mzeoli@COMPANY.PRI Name: mzeoli@COMPANY.PRI to mzeoli</LI-CODE>         Trying to understand where exactly the failure is when the same thing tries to happen in Livy. Unsure if the rules aren't adequate for what we're trying to do, or if it's simply not recognizing the rules / configs for some reason. DEBUG logs haven't been particular insightful, though we may have missed something in the haystack.   It's worth noting that we're in a multi-realm environment here. The users in question are part of COMPANY.PRI. the service principal and cluster itself is in BD.COMPANY.PRI. The rules seem to consider all of that fully (and our cluster itself has no problems over the last 2 yrs evaluating users that exist in COMPANY.PRI), so we're a bit confused and stuck. Any insight would be tremendously appreciated.   Thanks, Mike   ENV exports for livyblauser         <LI-CODE lang="markup">export SPARK_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/spark export SPARK_CONF_DIR=$SPARK_HOME/conf export JAVA_HOME=/usr/java/latest/ export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf</LI-CODE>         livy.conf  (all other items are remmed / default)         <LI-CODE lang="markup">livy.impersonation.enabled = false # for the execution of Livy itself livy.server.launch.kerberos.principal = livyblauser@COMPANY.PRI livy.server.launch.kerberos.keytab = /tmp/livyblauser.keytab # Livy has a built-in SPnego authentication support for HTTP requests with below configurations. livy.server.auth.type = kerberos livy.server.auth.kerberos.principal = HTTP/nydc-pblalivy01.bd.company.com@BD.COMPANY.PRI livy.server.auth.kerberos.keytab = /tmp/livyblauser.keytab # Use this keystore for the SSL certificate and key. livy.keystore = /opt/cloudera/security/jks/server.jks # Specify the keystore password. livy.keystore.password = xxxx # Specify the key password. livy.key-password = xxxx</LI-CODE>           Fri, 16 Sep 2022 14:36:47 GMT MikeZ 2022-09-16T14:36:47Z