Hello.   In CDH, the process of creating a parent pool that allowed dynamic child pools to be created at runtime was trivial -- simply mark the pool as Parent.  In CDP 7 I'm running into the following issue...   For a leaf pool  (lets say root.testpool), I edit pool attributes in the Yarn Queue Manager UI.   Under the "Dynamic Auto-Creation of Queue" section, I see....   "Is Auto Created Leaf Queue", which is unchecked and uneditable.  This makes sense since "root.testpool" was not auto created.   "Enable Dynamic Queue Creation' is unchecked but it also uneditable.  I feel this should be editable in order to define the "root.testpool" as supporting dynamic queues below it.   How do I enable dynamic queues for a given pool?   Thanks, M ... View more

Hello.   We are currently trying to get a proof-of-concept off the ground using Livy as a spark-shell multiplexor in our kerberized CDH 6.1 environment  (I realize Livy is formally supported in CDP 7, but our requirement atm is to get this working under 6 or create our own multiplexor to help reduce wait-time as spark shells start up).  We're running into problems getting Kerberos to properly map names via the auth_to_local rules.    We've completed the following: Deployed Livy to a clean, kerberized edge node on the cluster Created a service principal for Livy authentication  (see config below) Created a user principal for Livy server launch (see config below) Generated the keytabs needed to auth both of those principals Set the ENV vars needed for Livy to find the active hadoop config / libs  (see below) When we launch Livy, we get what seems to be a clean start, and it begins listening on tcp 8998 as expected.  The first time someone tries to hit that through a browser  (using Chrome, and we know it's passing spnego auth to Livy) we get the following:           20/05/21 13:03:12 WARN ServletHandler: /favicon.ico org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to mzeoli@COMPANY.PRI at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:347) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745)         The complaint about no rules being applied led us to review our auth_to_local rules in core-site.xml  (which is confirmed present in /etc/hadoop/conf), where we see the following local_to_auth rule:           <name>hadoop.security.auth_to_local</name> <value> RULE:[1:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$// RULE:[2:$1@$0](.*@\QBD.COMPANY.PRI\E$)s/@\QBD.COMPANY.PRI\E$// RULE:[1:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$// RULE:[2:$1@$0](.*@\QCOMPANY.PRI\E$)s/@\QCOMPANY.PRI\E$// DEFAULT </value>         If we try to check rule negotiation manually as the livyblauser user (same user that runs livy) from the same edge node, we see the following which looks functional and correct.         [livyblauser@nydc-pblalivy01 hadoop]$ hadoop org.apache.hadoop.security.HadoopKerberosName mzeoli@COMPANY.PRI Name: mzeoli@COMPANY.PRI to mzeoli         Trying to understand where exactly the failure is when the same thing tries to happen in Livy. Unsure if the rules aren't adequate for what we're trying to do, or if it's simply not recognizing the rules / configs for some reason. DEBUG logs haven't been particular insightful, though we may have missed something in the haystack.   It's worth noting that we're in a multi-realm environment here. The users in question are part of COMPANY.PRI. the service principal and cluster itself is in BD.COMPANY.PRI. The rules seem to consider all of that fully (and our cluster itself has no problems over the last 2 yrs evaluating users that exist in COMPANY.PRI), so we're a bit confused and stuck. Any insight would be tremendously appreciated.   Thanks, Mike   ENV exports for livyblauser         export SPARK_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/spark export SPARK_CONF_DIR=$SPARK_HOME/conf export JAVA_HOME=/usr/java/latest/ export HADOOP_HOME=/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p3380.937520/lib/hadoop export HADOOP_CONF_DIR=/etc/hadoop/conf         livy.conf  (all other items are remmed / default)         livy.impersonation.enabled = false # for the execution of Livy itself livy.server.launch.kerberos.principal = livyblauser@COMPANY.PRI livy.server.launch.kerberos.keytab = /tmp/livyblauser.keytab # Livy has a built-in SPnego authentication support for HTTP requests with below configurations. livy.server.auth.type = kerberos livy.server.auth.kerberos.principal = HTTP/nydc-pblalivy01.bd.company.com@BD.COMPANY.PRI livy.server.auth.kerberos.keytab = /tmp/livyblauser.keytab # Use this keystore for the SSL certificate and key. livy.keystore = /opt/cloudera/security/jks/server.jks # Specify the keystore password. livy.keystore.password = xxxx # Specify the key password. livy.key-password = xxxx           ... View more