Zeppelin with kerberized Livy error: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule

[#1]

Situation;

When running a Zeppelin notebook with the Livy interpreter, I get the following error;

Executed Command:

%livy.pyspark

print "1"

Error

Attempt recovered after RM restartAM Container for appattempt_1555963066902_0007_000001 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-04-23 10:36:30.617]Application application_1555963066902_0007 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is pc7 main : requested yarn user is pc7 org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name pc7@domain.local: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to pc7@domain.local at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847) at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710) at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660) at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.main(ContainerLocalizer.java:461) Caused by: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name pc7@domain.local: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to pc7@domain.local at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:232) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) at javax.security.auth.login.LoginContext.login(LoginContext.java:588) at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926) at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837) ... 4 more Caused by: java.lang.IllegalArgumentException: Illegal principal name pc7@domain.local: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to pc7@domain.local at org.apache.hadoop.security.User.<init>(User.java:51) at org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:230) ... 17 more Caused by: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to pc7@domain.local at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:401) at org.apache.hadoop.security.User.<init>(User.java:48) ... 18 more For more detailed output, check the application tracking page: http://hdpuat01.domain.local:8088/cluster/app/application_1555963066902_0007 Then click on links to logs of each attempt.

Environment:

HDP: 3.1

Ambari: 2.7.3

Kerberos Enabled

Services and configurations: All installed via Ambari

Hosts: 1 (UAT All in One environment)

What works:

Spark Interpretor

Livy command shown above with impersonation disabled

Can confirm exists:

pc7 account is able to kinit and obtain token

pc7 is able to access and alter hdfs/user/pc7 directory

hadoop.proxyuser.livy.groups = *

hadoop.proxyuser.livy.hosts = *

hadoop.proxyuser.zeppelin.groups = *

hadoop.proxyuser.zeppelin.hosts = *

hadoop.proxyuser.yarn.groups = *

hadoop.proxyuser.hdfs.groups = *

DEFAULT value in auth-to-local

Impersonation enabled (disabled now)

Livy.Superuser is zeppelin account without the @domain.local

Zeppelin.livy.principal is zeppelin account without the @domain.local

Livy keytab pertains to zeppelin user

Any help would be greatly appreciated!

[#2]

@Pierre Correia

It seems to be an issue with your auth_to_local runs best option before manually editing the auth_to_local is to regenerate the keytabs.

Tthe following clients installed hdfs,Yarn,spark client

Check the rules

HDFS-->Configs-->Advanced--> hadoop.security.auth_to_local

RULE:[1:$1@$0](ambari-qa-{cluster_name}@DOMAIN.LOCAL)s/.*/ambari-qa/
RULE:[1:$1@$0](hbase-{cluster_name}@DOMAIN.LOCAL)s/.*/hbase/
RULE:[1:$1@$0](hdfs-{cluster_name}@DOMAIN.LOCAL)s/.*/hdfs/
RULE:[1:$1@$0](spark-{cluster_name}@DOMAIN.LOCAL)s/.*/spark/
RULE:[1:$1@$0](zeppelin-{cluster_name}@DOMAIN.LOCAL)s/.*/zeppelin/
RULE:[1:$1@$0](.*@DOMAIN.LOCAL)s/@.*//
RULE:[2:$1@$0](amshbase@DOMAIN.LOCAL)s/.*/ams/
RULE:[2:$1@$0](amszk@DOMAIN.LOCAL)s/.*/ams/
RULE:[2:$1@$0](atlas@DOMAIN.LOCAL)s/.*/atlas/
RULE:[2:$1@$0](beacon@DOMAIN.LOCAL)s/.*/beacon/
RULE:[2:$1@$0](dn@DOMAIN.LOCAL)s/.*/hdfs/
RULE:[2:$1@$0](hbase@DOMAIN.LOCAL)s/.*/hbase/
RULE:[2:$1@$0](hive@DOMAIN.LOCAL)s/.*/hive/
RULE:[2:$1@$0](jhs@DOMAIN.LOCAL)s/.*/mapred/
RULE:[2:$1@$0](knox@DOMAIN.LOCAL)s/.*/knox/
RULE:[2:$1@$0](nifi@DOMAIN.LOCAL)s/.*/nifi/
RULE:[2:$1@$0](nm@DOMAIN.LOCAL)s/.*/yarn/
RULE:[2:$1@$0](nn@DOMAIN.LOCAL)s/.*/hdfs/
RULE:[2:$1@$0](oozie@DOMAIN.LOCAL)s/.*/oozie/
RULE:[2:$1@$0](rangeradmin@DOMAIN.LOCAL)s/.*/ranger/
RULE:[2:$1@$0](rangertagsync@DOMAIN.LOCAL)s/.*/rangertagsync/
RULE:[2:$1@$0](rangerusersync@DOMAIN.LOCAL)s/.*/rangerusersync/
RULE:[2:$1@$0](rm@DOMAIN.LOCAL)s/.*/yarn/
RULE:[2:$1@$0](yarn@DOMAIN.LOCAL)s/.*/yarn/
DEFAULT

Your rules shouldn't match but look like the above depending on the HDP components installed

[#3]

Thank you for the suggestions. The auth_to_local rules look very similar to the ones provided above, with respect to our domain. Ambari took care of creating them as services were added. Keytabs were also generated for all services, following a full service stack restart.

Unfortunately, I'm stuck with the same error;

main : run as user is pc7

main : requested yarn user is pc7

org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.IllegalArgumentException: Illegal principal name pc7@domain.local: org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to pc7@domain.local

[#4]

@Pierre Correia

I am sure I did send a response to this thread, please let me know,you need to create a user and associated keytabs. I documented it somewhere.

[#5]

Kerberos keytabs for all Ambari created users were regenerated. PC7 user also has a new kerberos token. Problem is still occurring.

[#6]

@Pierre Correia

Did you add a rule in the auth_to_local for you user pc7?

[#7]

I did not, but wouldn't the "RULE:[1:$1@$0](.*@DOMAIN.LOCAL)s/@.*//" take care of any username with that domain name? PC7 is a domain user, not a local.