Running into auth_to_local rule issue during Livy kerberos authentication

[#1]

Hello.

We are currently trying to get a proof-of-concept off the ground using Livy as a spark-shell multiplexor in our kerberized CDH 6.1 environment  (I realize Livy is formally supported in CDP 7, but our requirement atm is to get this working under 6 or create our own multiplexor to help reduce wait-time as spark shells start up).  We're running into problems getting Kerberos to properly map names via the auth_to_local rules. 

We've completed the following:

1. Deployed Livy to a clean, kerberized edge node on the cluster
2. Created a service principal for Livy authentication  (see config below)
3. Created a user principal for Livy server launch (see config below)
4. Generated the keytabs needed to auth both of those principals
5. Set the ENV vars needed for Livy to find the active hadoop config / libs  (see below)

When we launch Livy, we get what seems to be a clean start, and it begins listening on tcp 8998 as expected.  The first time someone tries to hit that through a browser  (using Chrome, and we know it's passing spnego auth to Livy) we get the following:

20/05/21 13:03:12 WARN ServletHandler: /favicon.ico
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mzeoli@COMPANY.PRI
at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347)
at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:539)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)

The complaint about no rules being applied led us to review our auth_to_local rules in core-site.xml  (which is confirmed present in /etc/hadoop/conf), where we see the following local_to_auth rule:

<name>hadoop.security.auth_to_local</name>
<value>
RULE:[1:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$//
RULE:[2:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$//
RULE:[1:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$//
RULE:[2:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$//
DEFAULT
</value>

If we try to check rule negotiation manually as the livyblauser user (same user that runs livy) from the same edge node, we see the following which looks functional and correct.

[livyblauser@nydc-pblalivy01 hadoop]$ hadoop org.apache.hadoop.security.HadoopKerberosName mzeoli@COMPANY.PRI
Name: mzeoli@COMPANY.PRI to mzeoli

Trying to understand where exactly the failure is when the same thing tries to happen in Livy. Unsure if the rules aren't adequate for what we're trying to do, or if it's simply not recognizing the rules / configs for some reason. DEBUG logs haven't been particular insightful, though we may have missed something in the haystack.

It's worth noting that we're in a multi-realm environment here. The users in question are part of COMPANY.PRI. the service principal and cluster itself is in BD.COMPANY.PRI. The rules seem to consider all of that fully (and our cluster itself has no problems over the last 2 yrs evaluating users that exist in COMPANY.PRI), so we're a bit confused and stuck. Any insight would be tremendously appreciated.

Thanks,
Mike

ENV exports for livyblauser

export SPARK_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export JAVA_HOME=/usr/java/latest/
export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf

livy.conf  (all other items are remmed / default)

livy.impersonation.enabled = false

# for the execution of Livy itself
livy.server.launch.kerberos.principal = livyblauser@COMPANY.PRI
livy.server.launch.kerberos.keytab = /tmp/livyblauser.keytab

# Livy has a built-in SPnego authentication support for HTTP requests with below configurations.
livy.server.auth.type = kerberos
livy.server.auth.kerberos.principal = HTTP/nydc-pblalivy01.bd.company.com@BD.COMPANY.PRI
livy.server.auth.kerberos.keytab = /tmp/livyblauser.keytab


# Use this keystore for the SSL certificate and key.
livy.keystore = /opt/cloudera/security/jks/server.jks

# Specify the keystore password.
livy.keystore.password = xxxx

# Specify the key password.
livy.key-password = xxxx

[#2]

Hello @MikeZ ,

thank you for raising your question about Livy on CDH6.

As you mentioned Livy is supported only in CDP [1] (Cloudera Data Platform 7), however it is not supported yet in CDH6. Usually a product is not supported until Cloudera finds it stable and production ready. Therefore we encourage you to evaluate CDP [2].

The "KerberosName$NoMatchingRule: No rules applied to mzeoli@COMPANY.PRI" exception usually occurs when the hadoop.security.auth_to_local rules does not provide a mapping to the local linux user (the user has to exist on the node that is extracted from the kerberos principal). Also make sure that the krb5.conf is readable by the user.

Please let us know if this information was helpful!

Kind regards:

Ferenc

[1] https://docs.cloudera.com/runtime/7.1.0/running-spark-applications/topics/spark-submitting-applicati...

[2] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-trial-download-information.html

[#3]

Thanks Bender.  I appreciate the response.  Just want to clarify a few things..

* Livy itself is running as livyblauser@COMPANY.PRI

* mzeoli@COMPANY.PRI is the user hitting the livy Web UI.

* Both livyblauser and mzeoli are AD accounts and have rights on the edge node livy is running on (nydc-pblalivy01, which is same box the HTTP service principal is for)

* Both have permission to read krb5.conf   (its world readable, though I'm not sure why / how something would be hitting krb5.conf as mzeoli, since mzeoli is just the web UI user and should not own any process.  Or perhaps I misunderstood you)

Given that this works...

[livyblauser@nydc-pblalivy01 hadoop]$ hadoop org.apache.hadoop.security.HadoopKerberosName mzeoli@COMPANY.PRI
Name: mzeoli@COMPANY.PRI to mzeoli

....It really feels like Livy isn't finding the rules it would expect to find, though I see the correct rules in /etc/hadoop/conf/core-site.xml.

<name>hadoop.security.auth_to_local</name>
<value>
RULE:[1:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$//
RULE:[2:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$//
RULE:[1:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$//
RULE:[2:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$//
DEFAULT
</value>

Thanks,

Mike

[#4]

Hello @MikeZ ,

thank you for getting back with your analysis. We do not recommend to customise the CDH releases, as it might lead to unforeseen instabilities that might be very difficult to debug.

Please consider to evaluate our CDP Trial release [1], in which the functionality you need is included and supported.

Best regards:

Ferenc

[1] https://docs.cloudera.com/cdpdc/7.0/installation/topics/cdpdc-trial-download-information.html

[#5]

Thanks Ferenc.  As mentioned, we cannot use CDP in the foreseeable future, nor are we customizing the CDH release in any real way -- simply adding Livy on a new edge node, which should work and should be relatively straightforward, but in this case is not.  Enterprise Support (which we pay for) is telling us to ask the Community for guidance.  So, here we are.

If anyone has any guidance in terms of actually getting past this local auth mapping issue, we would greatly appreciate it.

Best,

Mike